如何加速apply函數(shù)600倍的技巧

↓推薦關(guān)注↓

[ 引言 ] 雖然目前dask,cudf等包的出現(xiàn)，使得我們的數(shù)據(jù)處理大大得到了加速，但是并不是每個(gè)人都有比較好的gpu，非常多的朋友仍然還在使用pandas工具包，但有時(shí)候真的很無奈，pandas的許多問題我們都需要使用apply函數(shù)來進(jìn)行處理，而apply函數(shù)是非常慢的，本文我們就介紹如何加速apply函數(shù)600倍的技巧。

importpandasaspd
importnumpyasnp
df=pd.DataFrame(np.random.randint(0,11,size=(1000000,5)),columns=('a','b','c','d','e'))
deffunc(a,b,c,d,e):
ife==10:
returnc*d
elif(e=5):
returnc+d
elifereturna+b
%%time
df['new']=df.apply(lambdax:func(x['a'],x['b'],x['c'],x['d'],x['e']),axis=1)
CPUtimes:user17.9s,sys:301ms,total:18.2s
Walltime:18.4s

%%time
#!pipinstallswifter
importswifter
df['new']=df.swifter.apply(lambdax:func(x['a'],x['b'],x['c'],x['d'],x['e']),axis=1)
HBox(children=(HTML(value='DaskApply'),FloatProgress(value=0.0,max=16.0),HTML(value='')))

CPUtimes:user329ms,sys:240ms,total:569ms
Walltime:7.67s

• for循環(huán)；
• 列表處理；
• apply等操作

%%time
df['new']=df['c']*df['d']#defaultcasee==10
mask=df['e']'new']=df['c']+df['d']
mask=df['e']'new']=df['a']+df['b']
CPUtimes:user134ms,sys:149ms,total:283ms
Walltime:421ms

forcolin('a','b','c','d'):
df[col]=df[col].astype(np.int16)
%%time
df['new']=df['c']*df['d']#defaultcasee==10
mask=df['e']'new']=df['c']+df['d']
mask=df['e']'new']=df['a']+df['b']
CPUtimes:user71.3ms,sys:42.5ms,total:114ms
Walltime:116ms

• 此處先轉(zhuǎn)化為.values等價(jià)于轉(zhuǎn)化為numpy，這樣我們的向量化操作會(huì)更加快捷。

%%time
df['new']=df['c'].values*df['d'].values#defaultcasee==10
mask=df['e'].values'new']=df['c']+df['d']
mask=df['e'].values'new']=df['a']+df['b']
CPUtimes:user64.5ms,sys:12.5ms,total:77ms
Walltime:74.9ms

• Apply: 18.4 s
• Apply + Swifter: 7.67 s
• Pandas vectorizatoin: 421 ms
• Pandas vectorization + data types: 116 ms
• Pandas vectorization + values + data types: 74.9ms

審核編輯 ：李倩