Getting error while using compare in PySpark

[brahmavbr]

Hi,

I am using below code to create Profile reports and compare but when use compare command I am getting error. Need help to avoid this error and to proceed further.

Commands in Databricks:

%pip install ydata-profiling==4.0.0
2.
df=spark.sql("select * from limit 2000")
df.show()
3.
from ydata_profiling import ProfileReport

report = ProfileReport(df,
title='test1',
infer_dtypes=False,
interactions=None,
missing_diagrams=None,
correlations={"auto": {"calculate": False},
"pearson": {"calculate": False},
"spearman": {"calculate": False}})

report_html = report.to_html()
displayHTML(report_html)
/* it is displaying profile report properly. */

df1=df

report1 = ProfileReport(df1,
title='test1',
infer_dtypes=False,
interactions=None,
missing_diagrams=None,
correlations={"auto": {"calculate": False},
"pearson": {"calculate": False},
"spearman": {"calculate": False}})

report_html1 = report1.to_html()
displayHTML(report_html1)

comp_report=report.compare(report1,config=None)

This compare command throwing error as "AttributeError: 'DataFrame' object has no attribute 'loc'"

more error info:

AttributeError: 'DataFrame' object has no attribute 'loc'

AttributeError Traceback (most recent call last)
File :1
----> 1 comp_report=report.compare(report1,config=None)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-c8004347-4142-4d3e-b551-d46afdbaa0d9/lib/python3.9/site-packages/typeguard/init.py:1033, in typechecked..wrapper(*args, **kwargs)
1031 memo = _CallMemo(python_func, _localns, args=args, kwargs=kwargs)
1032 check_argument_types(memo)
-> 1033 retval = func(*args, **kwargs)
1034 try:
1035 check_return_type(retval, memo)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-c8004347-4142-4d3e-b551-d46afdbaa0d9/lib/python3.9/site-packages/ydata_profiling/profile_report.py:533, in ProfileReport.compare(self, other, config)
517 """Compare this report with another ProfileReport
518 Alias for:
519 ```
(...)
529 Comparison ProfileReport
530 """

Please help me why I am getting this error message, is it spark limitation or am I missing something.

Thank you.

Thanks,
Brahma

[fabclmnt]

Hi @brahmavbr ,

have you checked the pandas version that you have installed in that environment. It seems like it is not compatible with the requirements from the package.

Can you please share the details of your code environment? (Packages installed, versions, etc.)