ENH: df.grep(col,pat) and df.dselect(col,"expr")

wes, how would you feel about adding something like the following as
a Dataframe method? especially with method chaining I would find this
useful. Will add handling for datatypes and so on.

In [32]: from pandas.util.testing import makeCustomDataframe as mkdf
In [33]: def grep(df,col,pat):
    ...:     vals=df[col]
    ...:     return df.ix[[bool(re.search(pat,unicode(x))) for x in vals]]
In [34]: df=mkdf(20,2)
    ...: print df
C0       C_l0_g0 C_l0_g1  \
R0                         
R_l0_g0     R0C0    R0C1   
R_l0_g1     R1C0    R1C1   
R_l0_g10    R2C0    R2C1   
R_l0_g11    R3C0    R3C1   
R_l0_g12    R4C0    R4C1   
R_l0_g13    R5C0    R5C1   
R_l0_g14    R6C0    R6C1   
R_l0_g15    R7C0    R7C1   
R_l0_g16    R8C0    R8C1   
R_l0_g17    R9C0    R9C1   
R_l0_g18   R10C0   R10C1   
R_l0_g19   R11C0   R11C1   
R_l0_g2    R12C0   R12C1   
R_l0_g20   R13C0   R13C1   
R_l0_g3    R14C0   R14C1   
R_l0_g4    R15C0   R15C1   
R_l0_g5    R16C0   R16C1   
R_l0_g6    R17C0   R17C1   
R_l0_g7    R18C0   R18C1   
R_l0_g8    R19C0   R19C1   

In [35]: grep(df,"C_l0_g0","R.*6")
Out[35]: 
C0       C_l0_g0 C_l0_g1  \
R0                         
R_l0_g14    R6C0    R6C1   
R_l0_g5    R16C0   R16C1  

also

def dselect(self,col,f):
    f= eval("lambda x: " + f)
    return self.ix[[f(x) for x in self[col]]]

so this is possible:

In [13]: df=mkdf(20,2)
    ...: s=pd.Series(np.random.randint(1,100,len(df.index)),index=df.index,name="nums")
    ...: df=df.join(s)
    ...: print(df)
    ...: df.dselect("nums","x>50 or x<25").grep("C_l0_g0","[2|4|8]")
         C_l0_g0 C_l0_g1  nums  \
R0                               
R_l0_g0     R0C0    R0C1    55   
R_l0_g1     R1C0    R1C1    37   
R_l0_g10    R2C0    R2C1    61   
R_l0_g11    R3C0    R3C1    62   
R_l0_g12    R4C0    R4C1    75   
R_l0_g13    R5C0    R5C1    93   
R_l0_g14    R6C0    R6C1    31   
R_l0_g15    R7C0    R7C1    73   
R_l0_g16    R8C0    R8C1     6   
R_l0_g17    R9C0    R9C1    44   
R_l0_g18   R10C0   R10C1    97   
R_l0_g19   R11C0   R11C1    64   
R_l0_g2    R12C0   R12C1    39   
R_l0_g20   R13C0   R13C1    11   
R_l0_g3    R14C0   R14C1     5   
R_l0_g4    R15C0   R15C1    28   
R_l0_g5    R16C0   R16C1    25   
R_l0_g6    R17C0   R17C1    63   
R_l0_g7    R18C0   R18C1    21   
R_l0_g8    R19C0   R19C1    59   
Out[13]: 
         C_l0_g0 C_l0_g1  nums  \
R0                               
R_l0_g10    R2C0    R2C1    61   
R_l0_g12    R4C0    R4C1    75   
R_l0_g16    R8C0    R8C1     6   
R_l0_g3    R14C0   R14C1     5   
R_l0_g7    R18C0   R18C1    21  

not enough eval in pandas IMHO.

def grep(self,col_or_series,pat):
    if isinstance(col_or_series,pd.Series):
        col_or_series = col_or_series.name
    series=self[col_or_series]
    return self.ix[[bool(re.search(pat,unicode(x))) for x in series]]

def dselect(self,col_or_series,f=None):
    if isinstance(col_or_series,pd.Series):
        col_or_series = col_or_series.name
    series=self[col_or_series]
    if isinstance(f,basestring):
        f= eval("lambda x: " + f)
    return self.ix[[f(x) for x in series]]
pd.DataFrame.grep=grep
pd.DataFrame.dselect=dselect

dselect now accepts lambdas as well as eval snippets,
and you can pass in a dataframe column with tab completion,
not efficient but handy:

df.grep(df.colA,u'st\.[\s$]').dselect(df.colB,"x<=5").dselect(df.colC,lambda x: x[0] == myfilterval)

@wesm, if this has your blessing I'll round it out into a PR.

[jreback]

for selecting columns, does grep overlap with select/ filter?
what r use cases for:

• filter
• select
• grep

AFICT select/filter operate exclusively on index labels rather then data.
dselect and grep (maybe dgrep would be a better name) operate on data rather then indicies.

I have these monkey-patched onto pd.Dataframe at load-time and find them very useful.

An example use case:
a dataset from which you wish to get only rows that match a certain regexp in a column called "names".
Right now the best I'm aware of would be something like:

df.set_index("names",False,True).select(lambda x: bool(re.search("jerry.+",x[-1]))).reset_index(-1,drop=True)

edit: cleaned up the example. the bool around the re.search doesn't seem necessary but currently is.

IMO, This is very intuitive:

df.grep("names","jerry.+")

and with some fleshing out (multindex columns) could be a useful addition to core pandas.

[changhiskhan]

I'd prefer it if we didn't call it "grep". A more intuitive name like "search" or something would be better IMO. While you and I have no trouble understanding what "grepping" means, it's not the case for a lot of users.
And rather than having dselect, dgrep, or dsearch, whether to search the index or data or both should just be a keyword.

[jreback]

In addition, I think it makes sense to deprecate select (or make it a convenience method that uses filter; and promote usage of filter instead (which can do anything select can anyhow). my 2c

no strong opinion on naming here. whatever works.

rolling this into existing functions is a bit of a problem due to the existing
signatures.
select(crit,axis)
df.filter(items=None,like,regex=None)

• filter would need an extra axis arg, in which case it makes sense to filter on row
  labels as well as data. So axis would either have an additional "data" axis, which
  is incompatible with the rest of the API, or an additional argument which would really
  defeat the purpose of making the API more concise.
• Also , if an items argument were to be added to select it would have to be
  appended at the end so as not to break existing code:
  select(crit,axis,items)
df.filter(items=None,like,regex=None,axis)
  and so the "items" arg will apear in different locations for similar functions, That's really
  really bad design, I'd prefer giving up the whole thing then commiting a change like that.
• I could give up the implicit lambda in dselect and just enhance select to interpret
  a string crit arg as a regexp. But I think the implicit lambda is lovely.

pandas already seperates data from indicies on principle, would it be terrible
if the API reflected this as well?
There's a performance hit associated with data vs index operations,
because of the optimizations for indicies.
MIght be useful to differentiate the performant from the convenient.

related #1844 (comment)

related #2064 , i.e. "definitely more cowbell"

I find this functionaly extremely useful when working with "civic hacking" data, small data
sets where you want to filter against a name in a columns, and you really don't need to
convert them to an index.

Shouldn't pandas have (Explicitly slow and unvectorized in the general case) some functionality
for slicing and filtering according to data values? This keeps coming up as a need in data I work
with, and while monkey patching solves it for me, I'm guessing there are users who would benefit
from something like in core. the funky "string as implicit lambda body" can go away, It was just
an idea.

Any commiters +1 for something like this?

[hayd]

I definitely think something like this would be useful, however maybe it's useful to restrict to multiple columns at once (...though now I think about it you could just chain it).

I hadn't seen this thread but had been thinking something like this would be useful after DSM mentioned something about it on StackOverflow. (Note the terrible name choice.)

[jreback]

+1 for grep (search is ok too)

Also would consolidate filter and select (which filter just calls anyhow, and deprecate select) into just filter.

then filter for labels, and grep for data

We had this discussion above about trying to combine these, but I am +1 for keeping apart,
too much confusion, should be 1 way for labels, 1 for data, avoids ambiguity on what magic is happening

@y-p grep should have axis arg as well
(which I have also added into filter for my series refactor in any event)

very true about axis. I'm completely open to name changes and more functionalty,
just as long as the base form is concise.

Consider the funky lambda syntax gone, it's just a distraction and doesn't fit
in with the rest of the API.