Describe the bug
The 2-zone profiling exercise profiled full size placeholder_sandag and placeholder_psrc models, and found out that Accessibility Calculation is the memory peak for the two example 2-zone models. After tracing line by line, it turns out the following line gets triggered to access the xarray skim object, causing the memory spike. See more details in the presentation at Meeting 2023.03.28, the slides include a chart of memory comparison between sharrow turned on vs off.
|
|
|
result = self.dataset.iat(**x, _name=key) # iat strips data encoding |
|
# if 'digital_encoding' in self.dataset[key].attrs: |
|
# result = array_decode(result, self.dataset[key].attrs['digital_encoding']) |
|
|

Jeff pointed out that this line should not be triggered when it's running sharrow mode, something must be failing in the sharrow implementation. If implemented correctly, there should not be a memory spike when accessing skim object in sharrow mode.
Describe the bug
The 2-zone profiling exercise profiled full size placeholder_sandag and placeholder_psrc models, and found out that Accessibility Calculation is the memory peak for the two example 2-zone models. After tracing line by line, it turns out the following line gets triggered to access the xarray skim object, causing the memory spike. See more details in the presentation at Meeting 2023.03.28, the slides include a chart of memory comparison between sharrow turned on vs off.
activitysim/activitysim/core/skim_dataset.py
Lines 306 to 310 in d0776af
Jeff pointed out that this line should not be triggered when it's running sharrow mode, something must be failing in the sharrow implementation. If implemented correctly, there should not be a memory spike when accessing skim object in sharrow mode.