sampled from the caller object. | | See Also | -------- | numpy.random.choice: Generates a random sample from a given 1-D numpy | array. | | Notes | ----- | If `frac` > 1, `replacement` should be set to `True`. | | Examples | -------- | >>> df = pd.DataFrame({'num_legs': [2, 4, 8, 0], | ... 'num_wings': [2, 0, 0, 0], | ... 'num_specimen_seen': [10, 2, 1, 8]}, | ... index=['falcon', 'dog', 'spider', 'fish']) | >>> df | num_legs num_wings num_specimen_seen | falcon 2 2 10 | dog 4 0 2 | spider 8 0 1 | fish 0 0 8 | | Extract 3 random elements from the ``Series`` ``df['num_legs']``: | Note that we use `random_state` to ensure the reproducibility of | the examples. | | >>> df['num_legs'].sample(n=3, random_state=1) | fish 0 | spider 8 | falcon 2 | Name: num_legs, dtype: int64 | | A random 50% sample of the ``DataFrame`` with replacement: | | >>> df.sample(frac=0.5, replace=True, random_state=1) | num_legs num_wings num_specimen_seen | dog 4 0 2 | fish 0 0 8 | | An upsample sample of the ``DataFrame`` with replacement: | Note that `replace` parameter has to be `True` for `frac` parameter > 1. | | >>> df.sample(frac=2, replace=True, random_state=1) | num_legs num_wings num_specimen_seen | dog 4 0 2 | fish 0 0 8 | falcon 2 2 10 | falcon 2 2 10 | fish 0 0 8 | dog 4 0 2 | fish 0 0 8 | dog 4 0 2 | | Using a DataFrame column as weights. Rows with larger value in the | `num_specimen_seen` column are more likely to be sampled. | | >>> df.sample(n=2, weights='num_specimen_seen', random_state=1) | num_legs num_wings num_specimen_seen | falcon 2 2 10 | fish 0 0 8 | | set_axis(self, labels, axis=0, inplace=False) | Assign desired index to given axis. | | Indexes for column or row labels can be changed by assigning | a list-like or Index. | | .. versionchanged:: 0.21.0 | | The signature is now `labels` and `axis`, consistent with | the rest of pandas API. Previously, the `axis` and `labels` | arguments were respectively the first and second positional | arguments. | | Parameters | ---------- | labels : list-like, Index | The values for the new index. | | axis : {0 or 'index', 1 or 'columns'}, default 0 | The axis to update. The value 0 identifies the rows, and 1 | identifies the columns. | | inplace : bool, default False | Whether to return a new %(klass)s instance. | | Returns | ------- | renamed : %(klass)s or None | An object of same type as caller if inplace=False, None otherwise. | | See Also | -------- | DataFrame.rename_axis : Alter the name of the index or columns. | | Examples | -------- | **Series** | | >>> s = pd.Series([1, 2, 3]) | >>> s | 0 1 | 1 2 | 2 3 | dtype: int64 | | >>> s.set_axis(['a', 'b', 'c'], axis=0) | a 1 | b 2 | c 3 | dtype: int64 | | **DataFrame** | | >>> df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]}) | | Change the row labels. | | >>> df.set_axis(['a', 'b', 'c'], axis='index') | A B | a 1 4 | b 2 5 | c 3 6 | | Change the column labels. | | >>> df.set_axis(['I', 'II'], axis='columns') | I II | 0 1 4 | 1 2 5 | 2 3 6 | | Now, update the labels inplace. | | >>> df.set_axis(['i', 'ii'], axis='columns', inplace=True) | >>> df | i ii | 0 1 4 | 1 2 5 | 2 3 6 | | slice_shift(self: ~FrameOrSeries, periods: int = 1, axis=0) -> ~FrameOrSeries | Equivalent to `shift` without copying data. | | The shifted data will not include the dropped periods and the | shifted axis will be smaller than the original. | | Parameters | ---------- | periods : int | Number of periods to move, can be positive or negative. | | Returns | ------- | shifted : same type as caller | | Notes | ----- | While the `slice_shift` is faster than `shift`, you may pay for it | later during alignment. | | squeeze(self, axis=None) | Squeeze 1 dimensional axis objects into scalars. | | Series or DataFrames with a single element are squeezed to a scalar. | DataFrames with a single column or a single row are squeezed to a | Series. Otherwise the object is unchanged. | | This method is most useful when you don't know if your | object is a Series or DataFrame, but you do know it has just a single | column. In that case you can safely call `squeeze` to ensure you have a | Series. | | Parameters | ---------- | axis : {0 or 'index', 1 or 'columns', None}, default None | A specific axis to squeeze. By default, all length-1 axes are | squeezed. | | Returns | ------- | DataFrame, Series, or scalar | The projection after squeezing `axis` or all the axes. | | See Also | -------- | Series.iloc : Integer-location based indexing for selecting scalars. | DataFrame.iloc : Integer-location based indexing for selecting Series. | Series.to_frame : Inverse of DataFrame.squeeze for a | single-column DataFrame. | | Examples | -------- | >>> primes = pd.Series([2, 3, 5, 7]) | | Slicing might produce a Series with a single value: | | >>> even_primes = primes[primes % 2 == 0] | >>> even_primes | 0 2 | dtype: int64 | | >>> even_primes.squeeze() | 2 | | Squeezing objects with more than one value in every axis does nothing: | | >>> odd_primes = primes[primes % 2 == 1] | >>> odd_primes | 1 3 | 2 5 | 3 7 | dtype: int64 | | >>> odd_primes.squeeze() | 1 3 | 2 5 | 3 7 | dtype: int64 | | Squeezing is even more effective when used with DataFrames. | | >>> df = pd.DataFrame([[1, 2], [3, 4]], columns=['a', 'b']) | >>> df | a b | 0 1 2 | 1 3 4 | | Slicing a single column will produce a DataFrame with the columns | having only one value: | | >>> df_a = df[['a']] | >>> df_a | a | 0 1 | 1 3 | | So the columns can be squeezed down, resulting in a Series: | | >>> df_a.squeeze('columns') | 0 1 | 1 3 | Name: a, dtype: int64 | | Slicing a single row from a single column will produce a single | scalar DataFrame: | | >>> df_0a = df.loc[df.index < 1, ['a']] | >>> df_0a | a | 0 1 | | Squeezing the rows produces a single scalar Series: | | >>> df_0a.squeeze('rows') | a 1 | Name: 0, dtype: int64 | | Squeezing all axes will project directly into a scalar: | | >>> df_0a.squeeze() | 1 | | swapaxes(self: ~FrameOrSeries, axis1, axis2, copy=True) -> ~FrameOrSeries | Interchange axes and swap values axes appropriately. | | Returns | ------- | y : same as input | | tail(self: ~FrameOrSeries, n: int = 5) -> ~FrameOrSeries | Return the last `n` rows. | | This function returns last `n` rows from the object based on | position. It is useful for quickly verifying data, for example, | after sorting or appending rows. | | For negative values of `n`, this function returns all rows except | the first `n` rows, equivalent to ``df[n:]``. | | Parameters | ---------- | n : int, default 5 | Number of rows to select. | | Returns | ------- | type of caller | The last `n` rows of the caller object. | | See Also | -------- | DataFrame.head : The first `n` rows of the caller object. | | Examples | -------- | >>> df = pd.DataFrame({'animal': ['alligator', 'bee', 'falcon', 'lion', | ... 'monkey', 'parrot', 'shark', 'whale', 'zebra']}) | >>> df | animal | 0 alligator | 1 bee | 2 falcon | 3 lion | 4 monkey | 5 parrot | 6 shark | 7 whale | 8 zebra | | Viewing the last 5 lines | | >>> df.tail() | animal | 4 monkey | 5 parrot | 6 shark | 7 whale | 8 zebra | | Viewing the last `n` lines (three in this case) | | >>> df.tail(3) | animal | 6 shark | 7 whale | 8 zebra | | For negative values of `n` | | >>> df.tail(-3) | animal | 3 lion | 4 monkey | 5 parrot | 6 shark | 7 whale | 8 zebra | | to_clipboard(self, excel: bool = True, sep: Union[str, NoneType] = None, **kwargs) -> None | Copy object to the system clipboard. | | Write a text representation of object to the system clipboard. | This can be pasted into Excel, for example. | | Parameters | ---------- | excel : bool, default True | Produce output in a csv format for easy pasting into excel. | | - True, use the provided separator for csv pasting. | - False, write a string representation of the object to the clipboard. | | sep : str, default ``'\t'`` | Field delimiter. | **kwargs | These parameters will be passed to DataFrame.to_csv. | | See Also | -------- | DataFrame.to_csv : Write a DataFrame to a comma-separated values | (csv) file. | read_clipboard : Read text from clipboard and pass to read_table. | | Notes | ----- | Requirements for your platform. | | - Linux : `xclip`, or `xsel` (with `PyQt4` modules) | - Windows : none | - OS X : none | | Examples | -------- | Copy the contents of a DataFrame to the clipboard. | | >>> df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], columns=['A', 'B', 'C']) | >>> df.to_clipboard(sep=',') | ... # Wrote the following to the system clipboard: | ... # ,A,B,C | ... # 0,1,2,3 | ... # 1,4,5,6 | | We can omit the the index by passing the keyword `index` and setting | it to false. | | >>> df.to_clipboard(sep=',', index=False) | ... # Wrote the following to the system clipboard: | ... # A,B,C | ... # 1,2,3 | ... # 4,5,6 | | to_csv(self, path_or_buf: Union[str, pathlib.Path, IO[~AnyStr], NoneType] = None, sep: str = ',', na_rep: str = '', float_format: Union[str, NoneType] = None, columns: Union[Sequence[Union[Hashable, NoneType]], NoneType] = None, header: Union[bool, List[str]] = True, index: bool = True, index_label: Union[bool, str, Sequence[Union[Hashable, NoneType]], NoneType] = None, mode: str = 'w', encoding: Union[str, NoneType] = None, compression: Union[str, Mapping[str, str], NoneType] = 'infer', quoting: Union[int, NoneType] = None, quotechar: str = '"', line_terminator: Union[str, NoneType] = None, chunksize: Union[int, NoneType] = None, date_format: Union[str, NoneType] = None, doublequote: bool = True, escapechar: Union[str, NoneType] = None, decimal: Union[str, NoneType] = '.') -> Union[str, NoneType] | Write object to a comma-separated values (csv) file. | | .. versionchanged:: 0.24.0 | The order of arguments for Series was changed. | | Parameters | ---------- | path_or_buf : str or file handle, default None | File path or object, if None is provided the result is returned as | a string. If a file object is passed it should be opened with | `newline=''`, disabling universal newlines. | | .. versionchanged:: 0.24.0 | | Was previously named "path" for Series. | | sep : str, default ',' | String of length 1. Field delimiter for the output file. | na_rep : str, default '' | Missing data representation. | float_format : str, default None | Format string for floating point numbers. | columns : sequence, optional | Columns to write. | header : bool or list of str, default True | Write out the column names. If a list of strings is given it is | assumed to be aliases for the column names. | | .. versionchanged:: 0.24.0 | | Previously defaulted to False for Series. | | index : bool, default True | Write row names (index). | index_label : str or sequence, or False, default None | Column label for index column(s) if desired. If None is given, and | `header` and `index` are True, then the index names are used. A | sequence should be given if the object uses MultiIndex. If | False do not print fields for index names. Use index_label=False | for easier importing in R. | mode : str | Python write mode, default 'w'. | encoding : str, optional | A string representing the encoding to use in the output file, | defaults to 'utf-8'. | compression : str or dict, default 'infer' | If str, represents compression mode. If dict, value at 'method' is | the compression mode. Compression mode may be any of the following | possible values: {'infer', 'gzip', 'bz2', 'zip', 'xz', None}. If | compression mode is 'infer' and `path_or_buf` is path-like, then | detect compression mode from the following extensions: '.gz', | '.bz2', '.zip' or '.xz'. (otherwise no compression). If dict given | and mode is 'zip' or inferred as 'zip', other entries passed as | additional compression options. | | .. versionchanged:: 1.0.0 | | May now be a dict with key 'method' as compression mode | and other entries as additional compression options if | compression mode is 'zip'. | | quoting : optional constant from csv module | Defaults to csv.QUOTE_MINIMAL. If you have set a `float_format` | then floats are converted to strings and thus csv.QUOTE_NONNUMERIC | will treat them as non-numeric. | quotechar : str, default '\"' | String of length 1. Character used to quote fields. | line_terminator : str, optional | The newline character or character sequence to use in the output | file. Defaults to `os.linesep`, which depends on the OS in which | this method is called ('\n' for linux, '\r\n' for Windows, i.e.). | | .. versionchanged:: 0.24.0 | chunksize : int or None | Rows to write at a time. | date_format : str, default None | Format string for datetime objects. | doublequote : bool, default True | Control quoting of `quotechar` inside a field. | escapechar : str, default None | String of length 1. Character used to escape `sep` and `quotechar` | when appropriate. | decimal : str, default '.' | Character recognized as decimal separator. E.g. use ',' for | European data. | | Returns | ------- | None or str | If path_or_buf is None, returns the resulting csv format as a | string. Otherwise returns None. | | See Also | -------- | read_csv : Load a CSV file into a DataFrame. | to_excel : Write DataFrame to an Excel file. | | Examples | -------- | >>> df = pd.DataFrame({'name': ['Raphael', 'Donatello'], | ... 'mask': ['red', 'purple'], | ... 'weapon': ['sai', 'bo staff']}) | >>> df.to_csv(index=False) | 'name,mask,weapon\nRaphael,red,sai\nDonatello,purple,bo staff\n' | | Create 'out.zip' containing 'out.csv' | | >>> compression_opts = dict(method='zip', | ... archive_name='out.csv') # doctest: +SKIP | >>> df.to_csv('out.zip', index=False, | ... compression=compression_opts) # doctest: +SKIP | | to_excel(self, excel_writer, sheet_name='Sheet1', na_rep='', float_format=None, columns=None, header=True, index=True, index_label=None, startrow=0, startcol=0, engine=None, merge_cells=True, encoding=None, inf_rep='inf', verbose=True, freeze_panes=None) -> None | Write object to an Excel sheet. | | To write a single object to an Excel .xlsx file it is only necessary to | specify a target file name. To write to multiple sheets it is necessary to | create an `ExcelWriter` object with a target file name, and specify a sheet | in the file to write to. | | Multiple sheets may be written to by specifying unique `sheet_name`. | With all data written to the file it is necessary to save the changes. | Note that creating an `ExcelWriter` object with a file name that already | exists will result in the contents of the existing file being erased. | | Parameters | ---------- | excel_writer : str or ExcelWriter object | File path or existing ExcelWriter. | sheet_name : str, default 'Sheet1' | Name of sheet which will contain DataFrame. | na_rep : str, default '' | Missing data representation. | float_format : str, optional | Format string for floating point numbers. For example | ``float_format="%.2f"`` will format 0.1234 to 0.12. | columns : sequence or list of str, optional | Columns to write. | header : bool or list of str, default True | Write out the column names. If a list of string is given it is | assumed to be aliases for the column names. | index : bool, default True | Write row names (index). | index_label : str or sequence, optional | Column label for index column(s) if desired. If not specified, and | `header` and `index` are True, then the index names are used. A | sequence should be given if the DataFrame uses MultiIndex. | startrow : int, default 0 | Upper left cell row to dump data frame. | startcol : int, default 0 | Upper left cell column to dump data frame. | engine : str, optional | Write engine to use, 'openpyxl' or 'xlsxwriter'. You can also set this | via the options ``io.excel.xlsx.writer``, ``io.excel.xls.writer``, and | ``io.excel.xlsm.writer``. | merge_cells : bool, default True | Write MultiIndex and Hierarchical Rows as merged cells. | encoding : str, optional | Encoding of the resulting excel file. Only necessary for xlwt, | other writers support unicode natively. | inf_rep : str, default 'inf' | Representation for infinity (there is no native representation for | infinity in Excel). | verbose : bool, default True | Display more information in the error logs. | freeze_panes : tuple of int (length 2), optional | Specifies the one-based bottommost row and rightmost column that | is to be frozen. | | See Also | -------- | to_csv : Write DataFrame to a comma-separated values (csv) file. | ExcelWriter : Class for writing DataFrame objects into excel sheets. | read_excel : Read an Excel file into a pandas DataFrame. | read_csv : Read a comma-separated values (csv) file into DataFrame. | | Notes | ----- | For compatibility with :meth:`~DataFrame.to_csv`, | to_excel serializes lists and dicts to strings before writing. | | Once a workbook has been saved it is not possible write further data | without rewriting the whole workbook. | | Examples | -------- | | Create, write to and save a workbook: | | >>> df1 = pd.DataFrame([['a', 'b'], ['c', 'd']], | ... index=['row 1', 'row 2'], | ... columns=['col 1', 'col 2']) | >>> df1.to_excel("output.xlsx") # doctest: +SKIP | | To specify the sheet name: | | >>> df1.to_excel("output.xlsx", | ... sheet_name='Sheet_name_1') # doctest: +SKIP | | If you wish to write to more than one sheet in the workbook, it is | necessary to specify an ExcelWriter object: | | >>> df2 = df1.copy() | >>> with pd.ExcelWriter('output.xlsx') as writer: # doctest: +SKIP | ... df1.to_excel(writer, sheet_name='Sheet_name_1') | ... df2.to_excel(writer, sheet_name='Sheet_name_2') | | ExcelWriter can also be used to append to an existing Excel file: | | >>> with pd.ExcelWriter('output.xlsx', | ... mode='a') as writer: # doctest: +SKIP | ... df.to_excel(writer, sheet_name='Sheet_name_3') | | To set the library that is used to write the Excel file, | you can pass the `engine` keyword (the default engine is | automatically chosen depending on the file extension): | | >>> df1.to_excel('output1.xlsx', engine='xlsxwriter') # doctest: +SKIP | | to_hdf(self, path_or_buf, key: str, mode: str = 'a', complevel: Union[int, NoneType] = None, complib: Union[str, NoneType] = None, append: bool = False, format: Union[str, NoneType] = None, index: bool = True, min_itemsize: Union[int, Dict[str, int], NoneType] = None, nan_rep=None, dropna: Union[bool, NoneType] = None, data_columns: Union[List[str], NoneType] = None, errors: str = 'strict', encoding: str = 'UTF-8') -> None | Write the contained data to an HDF5 file using HDFStore. | | Hierarchical Data Format (HDF) is self-describing, allowing an | application to interpret the structure and contents of a file with | no outside information. One HDF file can hold a mix of related objects | which can be accessed as a group or as individual objects. | | In order to add another DataFrame or Series to an existing HDF file | please use append mode and a different a key. | | For more information see the :ref:`user guide <io.hdf5>`. | | Parameters | ---------- | path_or_buf : str or pandas.HDFStore | File path or HDFStore object. | key : str | Identifier for the group in the store. | mode : {'a', 'w', 'r+'}, default 'a' | Mode to open file: | | - 'w': write, a new file is created (an existing file with | the same name would be deleted). | - 'a': append, an existing file is opened for reading and | writing, and if the file does not exist it is created. | - 'r+': similar to 'a', but the file must already exist. | complevel : {0-9}, optional | Specifies a compression level for data. | A value of 0 disables compression. | complib : {'zlib', 'lzo', 'bzip2', 'blosc'}, default 'zlib' | Specifies the compression library to be used. | As of v0.20.2 these additional compressors for Blosc are supported | (default if no compressor specified: 'blosc:blosclz'): | {'blosc:blosclz', 'blosc:lz4', 'blosc:lz4hc', 'blosc:snappy', | 'blosc:zlib', 'blosc:zstd'}. | Specifying a compression library which is not available issues | a ValueError. | append : bool, default False | For Table formats, append the input data to the existing. | format : {'fixed', 'table', None}, default 'fixed' | Possible values: | | - 'fixed': Fixed format. Fast writing/reading. Not-appendable, | nor searchable. | - 'table': Table format. Write as a PyTables Table structure | which may perform worse but allow more flexible operations | like searching / selecting subsets of the data. | - If None, pd.get_option('io.hdf.default_format') is checked, | followed by fallback to "fixed" | errors : str, default 'strict' | Specifies how encoding and decoding errors are to be handled. | See the errors argument for :func:`open` for a full list | of options. | encoding : str, default "UTF-8" | min_itemsize : dict or int, optional | Map column names to minimum string sizes for columns. | nan_rep : Any, optional | How to represent null values as str. | Not allowed with append=True. | data_columns : list of columns or True, optional | List of columns to create as indexed data columns for on-disk | queries, or True to use all columns. By default only the axes | of the object are indexed. See :ref:`io.hdf5-query-data-columns`. | Applicable only to format='table'. | | See Also | -------- | DataFrame.read_hdf : Read from HDF file. | DataFrame.to_parquet : Write a DataFrame to the binary parquet format. | DataFrame.to_sql : Write to a sql table. | DataFrame.to_feather : Write out feather-format for DataFrames. | DataFrame.to_csv : Write out to a csv file. | | Examples | -------- | >>> df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]}, | ... index=['a', 'b', 'c']) | >>> df.to_hdf('data.h5', key='df', mode='w') | | We can add another object to the same file: | | >>> s = pd.Series([1, 2, 3, 4]) | >>> s.to_hdf('data.h5', key='s') | | Reading from HDF file: | | >>> pd.read_hdf('data.h5', 'df') | A B | a 1 4 | b 2 5 | c 3 6 | >>> pd.read_hdf('data.h5', 's') | 0 1 | 1 2 | 2 3 | 3 4 | dtype: int64 | | Deleting file with data: | | >>> import os | >>> os.remove('data.h5') | | to_json(self, path_or_buf: Union[str, pathlib.Path, IO[~AnyStr], NoneType] = None, orient: Union[str, NoneType] = None, date_format: Union[str, NoneType] = None, double_precision: int = 10, force_ascii: bool = True, date_unit: str = 'ms', default_handler: Union[Callable[[Any], Union[str, int, float, bool, List, Dict]], NoneType] = None, lines: bool = False, compression: Union[str, NoneType] = 'infer', index: bool = True, indent: Union[int, NoneType] = None) -> Union[str, NoneType] | Convert the object to a JSON string. | | Note NaN's and None will be converted to null and datetime objects | will be converted to UNIX timestamps. | | Parameters | ---------- | path_or_buf : str or file handle, optional | File path or object. If not specified, the result is returned as | a string. | orient : str | Indication of expected JSON string format. | | * Series: | | - default is 'index' | - allowed values are: {'split','records','index','table'}. | | * DataFrame: | | - default is 'columns' | - allowed values are: {'split', 'records', 'index', 'columns', | 'values', 'table'}. | | * The format of the JSON string: | | - 'split' : dict like {'index' -> [index], 'columns' -> [columns], | 'data' -> [values]} | - 'records' : list like [{column -> value}, ... , {column -> value}] | - 'index' : dict like {index -> {column -> value}} | - 'columns' : dict like {column -> {index -> value}} | - 'values' : just the values array | - 'table' : dict like {'schema': {schema}, 'data': {data}} | | Describing the data, where data component is like ``orient='records'``. | | .. versionchanged:: 0.20.0 | | date_format : {None, 'epoch', 'iso'} | Type of date conversion. 'epoch' = epoch milliseconds, | 'iso' = ISO8601. The default depends on the `orient`. For | ``orient='table'``, the default is 'iso'. For all other orients, | the default is 'epoch'. | double_precision : int, default 10 | The number of decimal places to use when encoding | floating point values. | force_ascii : bool, default True | Force encoded string to be ASCII. | date_unit : str, default 'ms' (milliseconds) | The time unit to encode to, governs timestamp and ISO8601 | precision. One of 's', 'ms', 'us', 'ns' for second, millisecond, | microsecond, and nanosecond respectively. | default_handler : callable, default None | Handler to call if object cannot otherwise be converted to a | suitable format for JSON. Should receive a single argument which is | the object to convert and return a serialisable object. | lines : bool, default False | If 'orient' is 'records' write out line delimited json format. Will | throw ValueError if incorrect 'orient' since others are not list | like. | | compression : {'infer', 'gzip', 'bz2', 'zip', 'xz', None} | | A string representing the compression to use in the output file, | only used when the first argument is a filename. By default, the | compression is inferred from the filename. | | .. versionadded:: 0.21.0 | .. versionchanged:: 0.24.0 | 'infer' option added and set to default | index : bool, default True | Whether to include the index values in the JSON string. Not | including the index (``index=False``) is only supported when | orient is 'split' or 'table'. | | .. versionadded:: 0.23.0 | | indent : int, optional | Length of whitespace used to indent each record. | | .. versionadded:: 1.0.0 | | Returns | ------- | None or str | If path_or_buf is None, returns the resulting json format as a | string. Otherwise returns None. | | See Also | -------- | read_json | | Notes | ----- | The behavior of ``indent=0`` varies from the stdlib, which does not | indent the output but does insert newlines. Currently, ``indent=0`` | and the default ``indent=None`` are equivalent in pandas, though this | may change in a future release. | | Examples | -------- | | >>> df = pd.DataFrame([['a', 'b'], ['c', 'd']], | ... index=['row 1', 'row 2'], | ... columns=['col 1', 'col 2']) | >>> df.to_json(orient='split') | '{"columns":["col 1","col 2"], | "index":["row 1","row 2"], | "data":[["a","b"],["c","d"]]}' | | Encoding/decoding a Dataframe using ``'records'`` formatted JSON. | Note that index labels are not preserved with this encoding. | | >>> df.to_json(orient='records') | '[{"col 1":"a","col 2":"b"},{"col 1":"c","col 2":"d"}]' | | Encoding/decoding a Dataframe using ``'index'`` formatted JSON: | | >>> df.to_json(orient='index') | '{"row 1":{"col 1":"a","col 2":"b"},"row 2":{"col 1":"c","col 2":"d"}}' | | Encoding/decoding a Dataframe using ``'columns'`` formatted JSON: | | >>> df.to_json(orient='columns') | '{"col 1":{"row 1":"a","row 2":"c"},"col 2":{"row 1":"b","row 2":"d"}}' | | Encoding/decoding a Dataframe using ``'values'`` formatted JSON: | | >>> df.to_json(orient='values') | '[["a","b"],["c","d"]]' | | Encoding with Table Schema | | >>> df.to_json(orient='table') | '{"schema": {"fields": [{"name": "index", "type": "string"}, | {"name": "col 1", "type": "string"}, | {"name": "col 2", "type": "string"}], | "primaryKey": "index", | "pandas_version": "0.20.0"}, | "data": [{"index": "row 1", "col 1": "a", "col 2": "b"}, | {"index": "row 2", "col 1": "c", "col 2": "d"}]}' | | to_latex(self, buf=None, columns=None, col_space=None, header=True, index=True, na_rep='NaN', formatters=None, float_format=None, sparsify=None, index_names=True, bold_rows=False, column_format=None, longtable=None, escape=None, encoding=None, decimal='.', multicolumn=None, multicolumn_format=None, multirow=None, caption=None, label=None) | Render object to a LaTeX tabular, longtable, or nested table/tabular. | | Requires ``\usepackage{booktabs}``. The output can be copy/pasted | into a main LaTeX document or read from an external file | with ``\input{table.tex}``. | | .. versionchanged:: 0.20.2 | Added to Series. | | .. versionchanged:: 1.0.0 | Added caption and label arguments. | | Parameters | ---------- | buf : str, Path or StringIO-like, optional, default None | Buffer to write to. If None, the output is returned as a string. | columns : list of label, optional | The subset of columns to write. Writes all columns by default. | col_space : int, optional | The minimum width of each column. | header : bool or list of str, default True | Write out the column names. If a list of strings is given, | it is assumed to be aliases for the column names. | index : bool, default True | Write row names (index). | na_rep : str, default 'NaN' | Missing data representation. | formatters : list of functions or dict of {str: function}, optional | Formatter functions to apply to columns' elements by position or | name. The result of each function must be a unicode string. | List must be of length equal to the number of columns. | float_format : one-parameter function or str, optional, default None | Formatter for floating point numbers. For example | ``float_format="%.2f"`` and ``float_format="{:0.2f}".format`` will | both result in 0.1234 being formatted as 0.12. | sparsify : bool, optional | Set to False for a DataFrame with a hierarchical index to print | every multiindex key at each row. By default, the value will be | read from the config module. | index_names : bool, default True | Prints the names of the indexes. | bold_rows : bool, default False | Make the row labels bold in the output. | column_format : str, optional | The columns format as specified in `LaTeX table format | <https://en.wikibooks.org/wiki/LaTeX/Tables>`__ e.g. 'rcl' for 3 | columns. By default, 'l' will be used for all columns except | columns of numbers, which default to 'r'. | longtable : bool, optional | By default, the value will be read from the pandas config | module. Use a longtable environment instead of tabular. Requires | adding a \usepackage{longtable} to your LaTeX preamble. | escape : bool, optional | By default, the value will be read from the pandas config | module. When set to False prevents from escaping latex special | characters in column names. | encoding : str, optional | A string representing the encoding to use in the output file, | defaults to 'utf-8'. | decimal : str, default '.' | Character recognized as decimal separator, e.g. ',' in Europe. | multicolumn : bool, default True | Use \multicolumn to enhance MultiIndex columns. | The default will be read from the config module. | multicolumn_format : str, default 'l' | The alignment for multicolumns, similar to `column_format` | The default will be read from the config module. | multirow : bool, default False | Use \multirow to enhance MultiIndex rows. Requires adding a | \usepackage{multirow} to your LaTeX preamble. Will print | centered labels (instead of top-aligned) across the contained | rows, separating groups via clines. The default will be read | from the pandas config module. | caption : str, optional | The LaTeX caption to be placed inside ``\caption{}`` in the output. | | .. versionadded:: 1.0.0 | | label : str, optional | The LaTeX label to be placed inside ``\label{}`` in the output. | This is used with ``\ref{}`` in the main ``.tex`` file. | | .. versionadded:: 1.0.0 | | Returns | ------- | str or None | If buf is None, returns the result as a string. Otherwise returns | None. | | See Also | -------- | DataFrame.to_string : Render a DataFrame to a console-friendly | tabular output. | DataFrame.to_html : Render a DataFrame as an HTML table. | | Examples | -------- | >>> df = pd.DataFrame({'name': ['Raphael', 'Donatello'], | ... 'mask': ['red', 'purple'], | ... 'weapon': ['sai', 'bo staff']}) | >>> print(df.to_latex(index=False)) # doctest: +NORMALIZE_WHITESPACE | \begin{tabular}{lll} | \toprule | name & mask & weapon \\ | \midrule | Raphael & red & sai \\ | Donatello & purple & bo staff \\ | \bottomrule | \end{tabular} | | to_pickle(self, path, compression: Union[str, NoneType] = 'infer', protocol: int = 5) -> None | Pickle (serialize) object to file. | | Parameters | ---------- | path : str | File path where the pickled object will be stored. | compression : {'infer', 'gzip', 'bz2', 'zip', 'xz', None}, default 'infer' | A string representing the compression to use in the output file. By | default, infers from the file extension in specified path. | protocol : int | Int which indicates which protocol should be used by the pickler, | default HIGHEST_PROTOCOL (see [1]_ paragraph 12.1.2). The possible | values are 0, 1, 2, 3, 4. A negative value for the protocol | parameter is equivalent to setting its value to HIGHEST_PROTOCOL. | | .. [1] https://docs.python.org/3/library/pickle.html. | .. versionadded:: 0.21.0. | | See Also | -------- | read_pickle : Load pickled pandas object (or any object) from file. | DataFrame.to_hdf : Write DataFrame to an HDF5 file. | DataFrame.to_sql : Write DataFrame to a SQL database. | DataFrame.to_parquet : Write a DataFrame to the binary parquet format. | | Examples | -------- | >>> original_df = pd.DataFrame({"foo": range(5), "bar": range(5, 10)}) | >>> original_df | foo bar | 0 0 5 | 1 1 6 | 2 2 7 | 3 3 8 | 4 4 9 | >>> original_df.to_pickle("./dummy.pkl") | | >>> unpickled_df = pd.read_pickle("./dummy.pkl") | >>> unpickled_df | foo bar | 0 0 5 | 1 1 6 | 2 2 7 | 3 3 8 | 4 4 9 | | >>> import os | >>> os.remove("./dummy.pkl") | | to_sql(self, name: str, con, schema=None, if_exists: str = 'fail', index: bool = True, index_label=None, chunksize=None, dtype=None, method=None) -> None | Write records stored in a DataFrame to a SQL database. | | Databases supported by SQLAlchemy [1]_ are supported. Tables can be | newly created, appended to, or overwritten. | | Parameters | ---------- | name : str | Name of SQL table. | con : sqlalchemy.engine.Engine or sqlite3.Connection | Using SQLAlchemy makes it possible to use any DB supported by that | library. Legacy support is provided for sqlite3.Connection objects. The user | is responsible for engine disposal and connection closure for the SQLAlchemy | connectable See `here <https://docs.sqlalchemy.org/en/13/core/connections.html>`_ | | schema : str, optional | Specify the schema (if database flavor supports this). If None, use | default schema. | if_exists : {'fail', 'replace', 'append'}, default 'fail' | How to behave if the table already exists. | | * fail: Raise a ValueError. | * replace: Drop the table before inserting new values. | * append: Insert new values to the existing table. | | index : bool, default True | Write DataFrame index as a column. Uses `index_label` as the column | name in the table. | index_label : str or sequence, default None | Column label for index column(s). If None is given (default) and | `index` is True, then the index names are used. | A sequence should be given if the DataFrame uses MultiIndex. | chunksize : int, optional | Specify the number of rows in each batch to be written at a time. | By default, all rows will be written at once. | dtype : dict or scalar, optional | Specifying the datatype for columns. If a dictionary is used, the | keys should be the column names and the values should be the | SQLAlchemy types or strings for the sqlite3 legacy mode. If a | scalar is provided, it will be applied to all columns. | method : {None, 'multi', callable}, optional | Controls the SQL insertion clause used: | | * None : Uses standard SQL ``INSERT`` clause (one per row). | * 'multi': Pass multiple values in a single ``INSERT`` clause. | * callable with signature ``(pd_table, conn, keys, data_iter)``. | | Details and a sample callable implementation can be found in the | section :ref:`insert method <io.sql.method>`. | | .. versionadded:: 0.24.0 | | Raises | ------ | ValueError | When the table already exists and `if_exists` is 'fail' (the | default). | | See Also | -------- | read_sql : Read a DataFrame from a table. | | Notes | ----- | Timezone aware datetime columns will be written as | ``Timestamp with timezone`` type with SQLAlchemy if supported by the | database. Otherwise, the datetimes will be stored as timezone unaware | timestamps local to the original timezone. | | .. versionadded:: 0.24.0 | | References | ---------- | .. [1] http://docs.sqlalchemy.org | .. [2] https://www.python.org/dev/peps/pep-0249/ | | Examples | -------- | | Create an in-memory SQLite database. | | >>> from sqlalchemy import create_engine | >>> engine = create_engine('sqlite://', echo=False) | | Create a table from scratch with 3 rows. | | >>> df = pd.DataFrame({'name' : ['User 1', 'User 2', 'User 3']}) | >>> df | name | 0 User 1 | 1 User 2 | 2 User 3 | | >>> df.to_sql('users', con=engine) | >>> engine.execute("SELECT * FROM users").fetchall() | [(0, 'User 1'), (1, 'User 2'), (2, 'User 3')] | | >>> df1 = pd.DataFrame({'name' : ['User 4', 'User 5']}) | >>> df1.to_sql('users', con=engine, if_exists='append') | >>> engine.execute("SELECT * FROM users").fetchall() | [(0, 'User 1'), (1, 'User 2'), (2, 'User 3'), | (0, 'User 4'), (1, 'User 5')] | | Overwrite the table with just ``df1``. | | >>> df1.to_sql('users', con=engine, if_exists='replace', | ... index_label='id') | >>> engine.execute("SELECT * FROM users").fetchall() | [(0, 'User 4'), (1, 'User 5')] | | Specify the dtype (especially useful for integers with missing values). | Notice that while pandas is forced to store the data as floating point, | the database supports nullable integers. When fetching the data with | Python, we get back integer scalars. | | >>> df = pd.DataFrame({"A": [1, None, 2]}) | >>> df | A | 0 1.0 | 1 NaN | 2 2.0 | | >>> from sqlalchemy.types import Integer | >>> df.to_sql('integers', con=engine, index=False, | ... dtype={"A": Integer()}) | | >>> engine.execute("SELECT * FROM integers").fetchall() | [(1,), (None,), (2,)] | | to_xarray(self) | Return an xarray object from the pandas object. | | Returns | ------- | xarray.DataArray or xarray.Dataset | Data in the pandas structure converted to Dataset if the object is | a DataFrame, or a DataArray if the object is a Series. | | See Also | -------- | DataFrame.to_hdf : Write DataFrame to an HDF5 file. | DataFrame.to_parquet : Write a DataFrame to the binary parquet format. | | Notes | ----- | See the `xarray docs <http://xarray.pydata.org/en/stable/>`__ | | Examples | -------- | >>> df = pd.DataFrame([('falcon', 'bird', 389.0, 2), | ... ('parrot', 'bird', 24.0, 2), | ... ('lion', 'mammal', 80.5, 4), | ... ('monkey', 'mammal', np.nan, 4)], | ... columns=['name', 'class', 'max_speed', | ... 'num_legs']) | >>> df | name class max_speed num_legs | 0 falcon bird 389.0 2 | 1 parrot bird 24.0 2 | 2 lion mammal 80.5 4 | 3 monkey mammal NaN 4 | | >>> df.to_xarray() | <xarray.Dataset> | Dimensions: (index: 4) | Coordinates: | * index (index) int64 0 1 2 3 | Data variables: | name (index) object 'falcon' 'parrot' 'lion' 'monkey' | class (index) object 'bird' 'bird' 'mammal' 'mammal' | max_speed (index) float64 389.0 24.0 80.5 nan | num_legs (index) int64 2 2 4 4 | | >>> df['max_speed'].to_xarray() | <xarray.DataArray 'max_speed' (index: 4)> | array([389. , 24. , 80.5, nan]) | Coordinates: | * index (index) int64 0 1 2 3 | | >>> dates = pd.to_datetime(['2018-01-01', '2018-01-01', | ... '2018-01-02', '2018-01-02']) | >>> df_multiindex = pd.DataFrame({'date': dates, | ... 'animal': ['falcon', 'parrot', | ... 'falcon', 'parrot'], | ... 'speed': [350, 18, 361, 15]}) | >>> df_multiindex = df_multiindex.set_index(['date', 'animal']) | | >>> df_multiindex | speed | date animal | 2018-01-01 falcon 350 | parrot 18 | 2018-01-02 falcon 361 | parrot 15 | | >>> df_multiindex.to_xarray() | <xarray.Dataset> | Dimensions: (animal: 2, date: 2) | Coordinates: | * date (date) datetime64[ns] 2018-01-01 2018-01-02 | * animal (animal) object 'falcon' 'parrot' | Data variables: | speed (date, animal) int64 350 18 361 15 | | truncate(self: ~FrameOrSeries, before=None, after=None, axis=None, copy: bool = True) -> ~FrameOrSeries | Truncate a Series or DataFrame before and after some index value. | | This is a useful shorthand for boolean indexing based on index | values above or below certain thresholds. | | Parameters | ---------- | before : date, str, int | Truncate all rows before this index value. | after : date, str, int | Truncate all rows after this index value. | axis : {0 or 'index', 1 or 'columns'}, optional | Axis to truncate. Truncates the index (rows) by default. | copy : bool, default is True, | Return a copy of the truncated section. | | Returns | ------- | type of caller | The truncated Series or DataFrame. | | See Also | -------- | DataFrame.loc : Select a subset of a DataFrame by label. | DataFrame.iloc : Select a subset of a DataFrame by position. | | Notes | ----- | If the index being truncated contains only datetime values, | `before` and `after` may be specified as strings instead of | Timestamps. | | Examples | -------- | >>> df = pd.DataFrame({'A': ['a', 'b', 'c', 'd', 'e'], | ... 'B': ['f', 'g', 'h', 'i', 'j'], | ... 'C': ['k', 'l', 'm', 'n', 'o']}, | ... index=[1, 2, 3, 4, 5]) | >>> df | A B C | 1 a f k | 2 b g l | 3 c h m | 4 d i n | 5 e j o | | >>> df.truncate(before=2, after=4) | A B C | 2 b g l | 3 c h m | 4 d i n | | The columns of a DataFrame can be truncated. | | >>> df.truncate(before="A", after="B", axis="columns") | A B | 1 a f | 2 b g | 3 c h | 4 d i | 5 e j | | For Series, only rows can be truncated. | | >>> df['A'].truncate(before=2, after=4) | 2 b | 3 c | 4 d | Name: A, dtype: object | | The index values in ``truncate`` can be datetimes or string | dates. | | >>> dates = pd.date_range('2016-01-01', '2016-02-01', freq='s') | >>> df = pd.DataFrame(index=dates, data={'A': 1}) | >>> df.tail() | A | 2016-01-31 23:59:56 1 | 2016-01-31 23:59:57 1 | 2016-01-31 23:59:58 1 | 2016-01-31 23:59:59 1 | 2016-02-01 00:00:00 1 | | >>> df.truncate(before=pd.Timestamp('2016-01-05'), | ... after=pd.Timestamp('2016-01-10')).tail() | A | 2016-01-09 23:59:56 1 | 2016-01-09 23:59:57 1 | 2016-01-09 23:59:58 1 | 2016-01-09 23:59:59 1 | 2016-01-10 00:00:00 1 | | Because the index is a DatetimeIndex containing only dates, we can | specify `before` and `after` as strings. They will be coerced to | Timestamps before truncation. | | >>> df.truncate('2016-01-05', '2016-01-10').tail() | A | 2016-01-09 23:59:56 1 | 2016-01-09 23:59:57 1 | 2016-01-09 23:59:58 1 | 2016-01-09 23:59:59 1 | 2016-01-10 00:00:00 1 | | Note that ``truncate`` assumes a 0 value for any unspecified time | component (midnight). This differs from partial string slicing, which | returns any partially matching dates. | | >>> df.loc['2016-01-05':'2016-01-10', :].tail() | A | 2016-01-10 23:59:55 1 | 2016-01-10 23:59:56 1 | 2016-01-10 23:59:57 1 | 2016-01-10 23:59:58 1 | 2016-01-10 23:59:59 1 | | tshift(self: ~FrameOrSeries, periods: int = 1, freq=None, axis=0) -> ~FrameOrSeries | Shift the time index, using the index's frequency if available. | | Parameters | ---------- | periods : int | Number of periods to move, can be positive or negative. | freq : DateOffset, timedelta, or str, default None | Increment to use from the tseries module | or time rule expressed as a string (e.g. 'EOM'). | axis : {0 or ‘index’, 1 or ‘columns’, None}, default 0 | Corresponds to the axis that contains the Index. | | Returns | ------- | shifted : Series/DataFrame | | Notes | ----- | If freq is not specified then tries to use the freq or inferred_freq | attributes of the index. If neither of those attributes exist, a | ValueError is thrown | | tz_convert(self: ~FrameOrSeries, tz, axis=0, level=None, copy: bool = True) -> ~FrameOrSeries | Convert tz-aware axis to target time zone. | | Parameters | ---------- | tz : str or tzinfo object | axis : the axis to convert | level : int, str, default None | If axis is a MultiIndex, convert a specific level. Otherwise | must be None. | copy : bool, default True | Also make a copy of the underlying data. | | Returns | ------- | %(klass)s | Object with time zone converted axis. | | Raises | ------ | TypeError | If the axis is tz-naive. | | tz_localize(self: ~FrameOrSeries, tz, axis=0, level=None, copy: bool = True, ambiguous='raise', nonexistent: str = 'raise') -> ~FrameOrSeries | Localize tz-naive index of a Series or DataFrame to target time zone. | | This operation localizes the Index. To localize the values in a | timezone-naive Series, use :meth:`Series.dt.tz_localize`. | | Parameters | ---------- | tz : str or tzinfo | axis : the axis to localize | level : int, str, default None | If axis ia a MultiIndex, localize a specific level. Otherwise | must be None. | copy : bool, default True | Also make a copy of the underlying data. | ambiguous : 'infer', bool-ndarray, 'NaT', default 'raise' | When clocks moved backward due to DST, ambiguous times may arise. | For example in Central European Time (UTC+01), when going from | 03:00 DST to 02:00 non-DST, 02:30:00 local time occurs both at | 00:30:00 UTC and at 01:30:00 UTC. In such a situation, the | `ambiguous` parameter dictates how ambiguous times should be | handled. | | - 'infer' will attempt to infer fall dst-transition hours based on | order | - bool-ndarray where True signifies a DST time, False designates | a non-DST time (note that this flag is only applicable for | ambiguous times) | - 'NaT' will return NaT where there are ambiguous times | - 'raise' will raise an AmbiguousTimeError if there are ambiguous | times. | nonexistent : str, default 'raise' | A nonexistent time does not exist in a particular timezone | where clocks moved forward due to DST. Valid values are: | | - 'shift_forward' will shift the nonexistent time forward to the | closest existing time | - 'shift_backward' will shift the nonexistent time backward to the | closest existing time | - 'NaT' will return NaT where there are nonexistent times | - timedelta objects will shift nonexistent times by the timedelta | - 'raise' will raise an NonExistentTimeError if there are | nonexistent times. | | .. versionadded:: 0.24.0 | | Returns | ------- | Series or DataFrame | Same type as the input. | | Raises | ------ | TypeError | If the TimeSeries is tz-aware and tz is not None. | | Examples | -------- | | Localize local times: | | >>> s = pd.Series([1], | ... index=pd.DatetimeIndex(['2018-09-15 01:30:00'])) | >>> s.tz_localize('CET') | 2018-09-15 01:30:00+02:00 1 | dtype: int64 | | Be careful with DST changes. When there is sequential data, pandas | can infer the DST time: | | >>> s = pd.Series(range(7), | ... index=pd.DatetimeIndex(['2018-10-28 01:30:00', | ... '2018-10-28 02:00:00', | ... '2018-10-28 02:30:00', | ... '2018-10-28 02:00:00', | ... '2018-10-28 02:30:00', | ... '2018-10-28 03:00:00', | ... '2018-10-28 03:30:00'])) | >>> s.tz_localize('CET', ambiguous='infer') | 2018-10-28 01:30:00+02:00 0 | 2018-10-28 02:00:00+02:00 1 | 2018-10-28 02:30:00+02:00 2 | 2018-10-28 02:00:00+01:00 3 | 2018-10-28 02:30:00+01:00 4 | 2018-10-28 03:00:00+01:00 5 | 2018-10-28 03:30:00+01:00 6 | dtype: int64 | | In some cases, inferring the DST is impossible. In such cases, you can | pass an ndarray to the ambiguous parameter to set the DST explicitly | | >>> s = pd.Series(range(3), | ... index=pd.DatetimeIndex(['2018-10-28 01:20:00', | ... '2018-10-28 02:36:00', | ... '2018-10-28 03:46:00'])) | >>> s.tz_localize('CET', ambiguous=np.array([True, True, False])) | 2018-10-28 01:20:00+02:00 0 | 2018-10-28 02:36:00+02:00 1 | 2018-10-28 03:46:00+01:00 2 | dtype: int64 | | If the DST transition causes nonexistent times, you can shift these | dates forward or backwards with a timedelta object or `'shift_forward'` | or `'shift_backwards'`. | >>> s = pd.Series(range(2), | ... index=pd.DatetimeIndex(['2015-03-29 02:30:00', | ... '2015-03-29 03:30:00'])) | >>> s.tz_localize('Europe/Warsaw', nonexistent='shift_forward') | 2015-03-29 03:00:00+02:00 0 | 2015-03-29 03:30:00+02:00 1 | dtype: int64 | >>> s.tz_localize('Europe/Warsaw', nonexistent='shift_backward') | 2015-03-29 01:59:59.999999999+01:00 0 | 2015-03-29 03:30:00+02:00 1 | dtype: int64 | >>> s.tz_localize('Europe/Warsaw', nonexistent=pd.Timedelta('1H')) | 2015-03-29 03:30:00+02:00 0 | 2015-03-29 03:30:00+02:00 1 | dtype: int64 | | where(self, cond, other=nan, inplace=False, axis=None, level=None, errors='raise', try_cast=False) | Replace values where the condition is False. | | Parameters | ---------- | cond : bool Series/DataFrame, array-like, or callable | Where `cond` is True, keep the original value. Where | False, replace with corresponding value from `other`. | If `cond` is callable, it is computed on the Series/DataFrame and | should return boolean Series/DataFrame or array. The callable must | not change input Series/DataFrame (though pandas doesn't check it). | other : scalar, Series/DataFrame, or callable | Entries where `cond` is False are replaced with | corresponding value from `other`. | If other is callable, it is computed on the Series/DataFrame and | should return scalar or Series/DataFrame. The callable must not | change input Series/DataFrame (though pandas doesn't check it). | inplace : bool, default False | Whether to perform the operation in place on the data. | axis : int, default None | Alignment axis if needed. | level : int, default None | Alignment level if needed. | errors : str, {'raise', 'ignore'}, default 'raise' | Note that currently this parameter won't affect | the results and will always coerce to a suitable dtype. | | - 'raise' : allow exceptions to be raised. | - 'ignore' : suppress exceptions. On error return original object. | | try_cast : bool, default False | Try to cast the result back to the input type (if possible). | | Returns | ------- | Same type as caller | | See Also | -------- | :func:`DataFrame.mask` : Return an object of same shape as | self. | | Notes | ----- | The where method is an application of the if-then idiom. For each | element in the calling DataFrame, if ``cond`` is ``True`` the | element is used; otherwise the corresponding element from the DataFrame | ``other`` is used. | | The signature for :func:`DataFrame.where` differs from | :func:`numpy.where`. Roughly ``df1.where(m, df2)`` is equivalent to | ``np.where(m, df1, df2)``. | | For further details and examples see the ``where`` documentation in | :ref:`indexing <indexing.where_mask>`. | | Examples | -------- | >>> s = pd.Series(range(5)) | >>> s.where(s > 0) | 0 NaN | 1 1.0 | 2 2.0 | 3 3.0 | 4 4.0 | dtype: float64 | | >>> s.mask(s > 0) | 0 0.0 | 1 NaN | 2 NaN | 3 NaN | 4 NaN | dtype: float64 | | >>> s.where(s > 1, 10) | 0 10 | 1 10 | 2 2 | 3 3 | 4 4 | dtype: int64 | | >>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=['A', 'B']) | >>> df | A B | 0 0 1 | 1 2 3 | 2 4 5 | 3 6 7 | 4 8 9 | >>> m = df % 3 == 0 | >>> df.where(m, -df) | A B | 0 0 -1 | 1 -2 3 | 2 -4 -5 | 3 6 -7 | 4 -8 9 | >>> df.where(m, -df) == np.where(m, df, -df) | A B | 0 True True | 1 True True | 2 True True | 3 True True | 4 True True | >>> df.where(m, -df) == df.mask(~m, -df) | A B | 0 True True | 1 True True | 2 True True | 3 True True | 4 True True | | xs(self, key, axis=0, level=None, drop_level: bool = True) | Return cross-section from the Series/DataFrame. | | This method takes a `key` argument to select data at a particular | level of a MultiIndex. | | Parameters | ---------- | key : label or tuple of label | Label contained in the index, or partially in a MultiIndex. | axis : {0 or 'index', 1 or 'columns'}, default 0 | Axis to retrieve cross-section on. | level : object, defaults to first n levels (n=1 or len(key)) | In case of a key partially contained in a MultiIndex, indicate | which levels are used. Levels can be referred by label or position. | drop_level : bool, default True | If False, returns object with same levels as self. | | Returns | ------- | Series or DataFrame | Cross-section from the original Series or DataFrame | corresponding to the selected index levels. | | See Also | -------- | DataFrame.loc : Access a group of rows and columns | by label(s) or a boolean array. | DataFrame.iloc : Purely integer-location based indexing | for selection by position. | | Notes | ----- | `xs` can not be used to set values. | | MultiIndex Slicers is a generic way to get/set values on | any level or levels. | It is a superset of `xs` functionality, see | :ref:`MultiIndex Slicers <advanced.mi_slicers>`. | | Examples | -------- | >>> d = {'num_legs': [4, 4, 2, 2], | ... 'num_wings': [0, 0, 2, 2], | ... 'class': ['mammal', 'mammal', 'mammal', 'bird'], | ... 'animal': ['cat', 'dog', 'bat', 'penguin'], | ... 'locomotion': ['walks', 'walks', 'flies', 'walks']} | >>> df = pd.DataFrame(data=d) | >>> df = df.set_index(['class', 'animal', 'locomotion']) | >>> df | num_legs num_wings | class animal locomotion | mammal cat walks 4 0 | dog walks 4 0 | bat flies 2 2 | bird penguin walks 2 2 | | Get values at specified index | | >>> df.xs('mammal') | num_legs num_wings | animal locomotion | cat walks 4 0 | dog walks 4 0 | bat flies 2 2 | | Get values at several indexes | | >>> df.xs(('mammal', 'dog')) | num_legs num_wings | locomotion | walks 4 0 | | Get values at specified index and level | | >>> df.xs('cat', level=1) | num_legs num_wings | class locomotion | mammal walks 4 0 | | Get values at several indexes and levels | | >>> df.xs(('bird', 'walks'), | ... level=[0, 'locomotion']) | num_legs num_wings | animal | penguin 2 2 | | Get values at specified column and axis | | >>> df.xs('num_wings', axis=1) | class animal locomotion | mammal cat walks 0 | dog walks 0 | bat flies 2 | bird penguin walks 2 | Name: num_wings, dtype: int64 | | ---------------------------------------------------------------------- | Data descriptors inherited from pandas.core.generic.NDFrame: | | attrs | Dictionary of global attributes on this object. | | .. warning:: | | attrs is experimental and may change without warning. | | ---------------------------------------------------------------------- | Methods inherited from pandas.core.base.PandasObject: | | __sizeof__(self) | Generates the total memory usage for an object that returns | either a value or Series of values | | ---------------------------------------------------------------------- | Methods inherited from pandas.core.accessor.DirNamesMixin: | | __dir__(self) | Provide method name lookup and completion. | | Notes | ----- | Only provide 'public' methods. | | ---------------------------------------------------------------------- | Readonly properties inherited from pandas.core.indexing.IndexingMixin: | | at | Access a single value for a row/column label pair. | | Similar to ``loc``, in that both provide label-based lookups. Use | ``at`` if you only need to get or set a single value in a DataFrame | or Series. | | Raises | ------ | KeyError | If 'label' does not exist in DataFrame. | | See Also | -------- | DataFrame.iat : Access a single value for a row/column pair by integer | position. | DataFrame.loc : Access a group of rows and columns by label(s). | Series.at : Access a single value using a label. | | Examples | -------- | >>> df = pd.DataFrame([[0, 2, 3], [0, 4, 1], [10, 20, 30]], | ... index=[4, 5, 6], columns=['A', 'B', 'C']) | >>> df | A B C | 4 0 2 3 | 5 0 4 1 | 6 10 20 30 | | Get value at specified row/column pair | | >>> df.at[4, 'B'] | 2 | | Set value at specified row/column pair | | >>> df.at[4, 'B'] = 10 | >>> df.at[4, 'B'] | 10 | | Get value within a Series | | >>> df.loc[5].at['B'] | 4 | | iat | Access a single value for a row/column pair by integer position. | | Similar to ``iloc``, in that both provide integer-based lookups. Use | ``iat`` if you only need to get or set a single value in a DataFrame | or Series. | | Raises | ------ | IndexError | When integer position is out of bounds. | | See Also | -------- | DataFrame.at : Access a single value for a row/column label pair. | DataFrame.loc : Access a group of rows and columns by label(s). | DataFrame.iloc : Access a group of rows and columns by integer position(s). | | Examples | -------- | >>> df = pd.DataFrame([[0, 2, 3], [0, 4, 1], [10, 20, 30]], | ... columns=['A', 'B', 'C']) | >>> df | A B C | 0 0 2 3 | 1 0 4 1 | 2 10 20 30 | | Get value at specified row/column pair | | >>> df.iat[1, 2] | 1 | | Set value at specified row/column pair | | >>> df.iat[1, 2] = 10 | >>> df.iat[1, 2] | 10 | | Get value within a series | | >>> df.loc[0].iat[1] | 2 | | iloc | Purely integer-location based indexing for selection by position. | | ``.iloc[]`` is primarily integer position based (from ``0`` to | ``length-1`` of the axis), but may also be used with a boolean | array. | | Allowed inputs are: | | - An integer, e.g. ``5``. | - A list or array of integers, e.g. ``[4, 3, 0]``. | - A slice object with ints, e.g. ``1:7``. | - A boolean array. | - A ``callable`` function with one argument (the calling Series or | DataFrame) and that returns valid output for indexing (one of the above). | This is useful in method chains, when you don't have a reference to the | calling object, but would like to base your selection on some value. | | ``.iloc`` will raise ``IndexError`` if a requested indexer is | out-of-bounds, except *slice* indexers which allow out-of-bounds | indexing (this conforms with python/numpy *slice* semantics). | | See more at :ref:`Selection by Position <indexing.integer>`. | | See Also | -------- | DataFrame.iat : Fast integer location scalar accessor. | DataFrame.loc : Purely label-location based indexer for selection by label. | Series.iloc : Purely integer-location based indexing for | selection by position. | | Examples | -------- | | >>> mydict = [{'a': 1, 'b': 2, 'c': 3, 'd': 4}, | ... {'a': 100, 'b': 200, 'c': 300, 'd': 400}, | ... {'a': 1000, 'b': 2000, 'c': 3000, 'd': 4000 }] | >>> df = pd.DataFrame(mydict) | >>> df | a b c d | 0 1 2 3 4 | 1 100 200 300 400 | 2 1000 2000 3000 4000 | | **Indexing just the rows** | | With a scalar integer. | | >>> type(df.iloc[0]) | <class 'pandas.core.series.Series'> | >>> df.iloc[0] | a 1 | b 2 | c 3 | d 4 | Name: 0, dtype: int64 | | With a list of integers. | | >>> df.iloc[[0]] | a b c d | 0 1 2 3 4 | >>> type(df.iloc[[0]]) | <class 'pandas.core.frame.DataFrame'> | | >>> df.iloc[[0, 1]] | a b c d | 0 1 2 3 4 | 1 100 200 300 400 | | With a `slice` object. | | >>> df.iloc[:3] | a b c d | 0 1 2 3 4 | 1 100 200 300 400 | 2 1000 2000 3000 4000 | | With a boolean mask the same length as the index. | | >>> df.iloc[[True, False, True]] | a b c d | 0 1 2 3 4 | 2 1000 2000 3000 4000 | | With a callable, useful in method chains. The `x` passed | to the ``lambda`` is the DataFrame being sliced. This selects | the rows whose index label even. | | >>> df.iloc[lambda x: x.index % 2 == 0] | a b c d | 0 1 2 3 4 | 2 1000 2000 3000 4000 | | **Indexing both axes** | | You can mix the indexer types for the index and columns. Use ``:`` to | select the entire axis. | | With scalar integers. | | >>> df.iloc[0, 1] | 2 | | With lists of integers. | | >>> df.iloc[[0, 2], [1, 3]] | b d | 0 2 4 | 2 2000 4000 | | With `slice` objects. | | >>> df.iloc[1:3, 0:3] | a b c | 1 100 200 300 | 2 1000 2000 3000 | | With a boolean array whose length matches the columns. | | >>> df.iloc[:, [True, False, True, False]] | a c | 0 1 3 | 1 100 300 | 2 1000 3000 | | With a callable function that expects the Series or DataFrame. | | >>> df.iloc[:, lambda df: [0, 2]] | a c | 0 1 3 | 1 100 300 | 2 1000 3000 | | loc | Access a group of rows and columns by label(s) or a boolean array. | | ``.loc[]`` is primarily label based, but may also be used with a | boolean array. | | Allowed inputs are: | | - A single label, e.g. ``5`` or ``'a'``, (note that ``5`` is | interpreted as a *label* of the index, and **never** as an | integer position along the index). | - A list or array of labels, e.g. ``['a', 'b', 'c']``. | - A slice object with labels, e.g. ``'a':'f'``. | | .. warning:: Note that contrary to usual python slices, **both** the | start and the stop are included | | - A boolean array of the same length as the axis being sliced, | e.g. ``[True, False, True]``. | - A ``callable`` function with one argument (the calling Series or | DataFrame) and that returns valid output for indexing (one of the above) | | See more at :ref:`Selection by Label <indexing.label>` | | Raises | ------ | KeyError | If any items are not found. | | See Also | -------- | DataFrame.at : Access a single value for a row/column label pair. | DataFrame.iloc : Access group of rows and columns by integer position(s). | DataFrame.xs : Returns a cross-section (row(s) or column(s)) from the | Series/DataFrame. | Series.loc : Access group of values using labels. | | Examples | -------- | **Getting values** | | >>> df = pd.DataFrame([[1, 2], [4, 5], [7, 8]], | ... index=['cobra', 'viper', 'sidewinder'], | ... columns=['max_speed', 'shield']) | >>> df | max_speed shield | cobra 1 2 | viper 4 5 | sidewinder 7 8 | | Single label. Note this returns the row as a Series. | | >>> df.loc['viper'] | max_speed 4 | shield 5 | Name: viper, dtype: int64 | | List of labels. Note using ``[[]]`` returns a DataFrame. | | >>> df.loc[['viper', 'sidewinder']] | max_speed shield | viper 4 5 | sidewinder 7 8 | | Single label for row and column | | >>> df.loc['cobra', 'shield'] | 2 | | Slice with labels for row and single label for column. As mentioned | above, note that both the start and stop of the slice are included. | | >>> df.loc['cobra':'viper', 'max_speed'] | cobra 1 | viper 4 | Name: max_speed, dtype: int64 | | Boolean list with the same length as the row axis | | >>> df.loc[[False, False, True]] | max_speed shield | sidewinder 7 8 | | Conditional that returns a boolean Series | | >>> df.loc[df['shield'] > 6] | max_speed shield | sidewinder 7 8 | | Conditional that returns a boolean Series with column labels specified | | >>> df.loc[df['shield'] > 6, ['max_speed']] | max_speed | sidewinder 7 | | Callable that returns a boolean Series | | >>> df.loc[lambda df: df['shield'] == 8] | max_speed shield | sidewinder 7 8 | | **Setting values** | | Set value for all items matching the list of labels | | >>> df.loc[['viper', 'sidewinder'], ['shield']] = 50 | >>> df | max_speed shield | cobra 1 2 | viper 4 50 | sidewinder 7 50 | | Set value for an entire row | | >>> df.loc['cobra'] = 10 | >>> df | max_speed shield | cobra 10 10 | viper 4 50 | sidewinder 7 50 | | Set value for an entire column | | >>> df.loc[:, 'max_speed'] = 30 | >>> df | max_speed shield | cobra 30 10 | viper 30 50 | sidewinder 30 50 | | Set value for rows matching callable condition | | >>> df.loc[df['shield'] > 35] = 0 | >>> df | max_speed shield | cobra 30 10 | viper 0 0 | sidewinder 0 0 | | **Getting values on a DataFrame with an index that has integer labels** | | Another example using integers for the index | | >>> df = pd.DataFrame([[1, 2], [4, 5], [7, 8]], | ... index=[7, 8, 9], columns=['max_speed', 'shield']) | >>> df | max_speed shield | 7 1 2 | 8 4 5 | 9 7 8 | | Slice with integer labels for rows. As mentioned above, note that both | the start and stop of the slice are included. | | >>> df.loc[7:9] | max_speed shield | 7 1 2 | 8 4 5 | 9 7 8 | | **Getting values with a MultiIndex** | | A number of examples using a DataFrame with a MultiIndex | | >>> tuples = [ | ... ('cobra', 'mark i'), ('cobra', 'mark ii'), | ... ('sidewinder', 'mark i'), ('sidewinder', 'mark ii'), | ... ('viper', 'mark ii'), ('viper', 'mark iii') | ... ] | >>> index = pd.MultiIndex.from_tuples(tuples) | >>> values = [[12, 2], [0, 4], [10, 20], | ... [1, 4], [7, 1], [16, 36]] | >>> df = pd.DataFrame(values, columns=['max_speed', 'shield'], index=index) | >>> df | max_speed shield | cobra mark i 12 2 | mark ii 0 4 | sidewinder mark i 10 20 | mark ii 1 4 | viper mark ii 7 1 | mark iii 16 36 | | Single label. Note this returns a DataFrame with a single index. | | >>> df.loc['cobra'] | max_speed shield | mark i 12 2 | mark ii 0 4 | | Single index tuple. Note this returns a Series. | | >>> df.loc[('cobra', 'mark ii')] | max_speed 0 | shield 4 | Name: (cobra, mark ii), dtype: int64 | | Single label for row and column. Similar to passing in a tuple, this | returns a Series. | | >>> df.loc['cobra', 'mark i'] | max_speed 12 | shield 2 | Name: (cobra, mark i), dtype: int64 | | Single tuple. Note using ``[[]]`` returns a DataFrame. | | >>> df.loc[[('cobra', 'mark ii')]] | max_speed shield | cobra mark ii 0 4 | | Single tuple for the index with a single label for the column | | >>> df.loc[('cobra', 'mark i'), 'shield'] | 2 | | Slice from index tuple to single label | | >>> df.loc[('cobra', 'mark i'):'viper'] | max_speed shield | cobra mark i 12 2 | mark ii 0 4 | sidewinder mark i 10 20 | mark ii 1 4 | viper mark ii 7 1 | mark iii 16 36 | | Slice from index tuple to index tuple | | >>> df.loc[('cobra', 'mark i'):('viper', 'mark ii')] | max_speed shield | cobra mark i 12 2 | mark ii 0 4 | sidewinder mark i 10 20 | mark ii 1 4 | viper mark ii 7 1
# However, using help function with a non-existent object will trigger a NameError as usual
help(Series)
--------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-99-06f975387760> in <module> ----> 1 help(Series) NameError: name 'Series' is not defined
# We can see a list of available functions, methods and attributes of an object using the dir() function
# When you scroll down you will see that we can call argmax, argmin, append, apply and
# many accessible functions to create and manipulate pd.Series
dir(pd.Series)
['T', '_AXIS_ALIASES', '_AXIS_IALIASES', '_AXIS_LEN', '_AXIS_NAMES', '_AXIS_NUMBERS', '_AXIS_ORDERS', '_AXIS_REVERSED', '_HANDLED_TYPES', '__abs__', '__add__', '__and__', '__annotations__', '__array__', '__array_priority__', '__array_ufunc__', '__array_wrap__', '__bool__', '__class__', '__contains__', '__copy__', '__deepcopy__', '__delattr__', '__delitem__', '__dict__', '__dir__', '__div__', '__divmod__', '__doc__', '__eq__', '__finalize__', '__float__', '__floordiv__', '__format__', '__ge__', '__getattr__', '__getattribute__', '__getitem__', '__getstate__', '__gt__', '__hash__', '__iadd__', '__iand__', '__ifloordiv__', '__imod__', '__imul__', '__init__', '__init_subclass__', '__int__', '__invert__', '__ior__', '__ipow__', '__isub__', '__iter__', '__itruediv__', '__ixor__', '__le__', '__len__', '__long__', '__lt__', '__matmul__', '__mod__', '__module__', '__mul__', '__ne__', '__neg__', '__new__', '__nonzero__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdiv__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rmatmul__', '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__setitem__', '__setstate__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__weakref__', '__xor__', '_accessors', '_add_numeric_operations', '_add_series_or_dataframe_operations', '_agg_by_level', '_agg_examples_doc', '_agg_see_also_doc', '_aggregate', '_aggregate_multiple_funcs', '_align_frame', '_align_series', '_binop', '_box_item_values', '_builtin_table', '_can_hold_na', '_check_inplace_setting', '_check_is_chained_assignment_possible', '_check_label_or_level_ambiguity', '_check_setitem_copy', '_clear_item_cache', '_clip_with_one_bound', '_clip_with_scalar', '_consolidate', '_consolidate_inplace', '_construct_axes_dict', '_construct_axes_dict_from', '_construct_axes_from_arguments', '_constructor', '_constructor_expanddim', '_constructor_sliced', '_convert', '_convert_dtypes', '_create_indexer', '_cython_table', '_deprecations', '_dir_additions', '_dir_deletions', '_drop_axis', '_drop_labels_or_levels', '_find_valid_index', '_from_axes', '_get_axis', '_get_axis_name', '_get_axis_number', '_get_axis_resolvers', '_get_block_manager_axis', '_get_bool_data', '_get_cacher', '_get_cleaned_column_resolvers', '_get_cython_func', '_get_index_resolvers', '_get_item_cache', '_get_label_or_level_values', '_get_numeric_data', '_get_value', '_get_values', '_get_values_tuple', '_get_with', '_gotitem', '_iget_item_cache', '_index', '_indexed_same', '_info_axis', '_info_axis_name', '_info_axis_number', '_init_dict', '_init_mgr', '_internal_get_values', '_internal_names', '_internal_names_set', '_is_builtin_func', '_is_cached', '_is_copy', '_is_datelike_mixed_type', '_is_label_or_level_reference', '_is_label_reference', '_is_level_reference', '_is_mixed_type', '_is_numeric_mixed_type', '_is_view', '_ix', '_ixs', '_map_values', '_maybe_cache_changed', '_maybe_update_cacher', '_metadata', '_ndarray_values', '_needs_reindex_multi', '_obj_with_exclusions', '_protect_consolidate', '_reduce', '_reindex_axes', '_reindex_indexer', '_reindex_multi', '_reindex_with_indexers', '_repr_data_resource_', '_repr_latex_', '_reset_cache', '_reset_cacher', '_selected_obj', '_selection', '_selection_list', '_selection_name', '_set_as_cached', '_set_axis', '_set_axis_name', '_set_is_copy', '_set_item', '_set_labels', '_set_name', '_set_subtyp', '_set_value', '_set_values', '_set_with', '_set_with_engine', '_setup_axes', '_slice', '_stat_axis', '_stat_axis_name', '_stat_axis_number', '_take_with_is_copy', '_to_dict_of_blocks', '_try_aggregate_string_function', '_typ', '_unpickle_series_compat', '_update_inplace', '_validate_dtype', '_values', '_where', '_xs', 'abs', 'add', 'add_prefix', 'add_suffix', 'agg', 'aggregate', 'align', 'all', 'any', 'append', 'apply', 'argmax', 'argmin', 'argsort', 'array', 'asfreq', 'asof', 'astype', 'at', 'at_time', 'attrs', 'autocorr', 'axes', 'between', 'between_time', 'bfill', 'bool', 'cat', 'clip', 'combine', 'combine_first', 'convert_dtypes', 'copy', 'corr', 'count', 'cov', 'cummax', 'cummin', 'cumprod', 'cumsum', 'describe', 'diff', 'div', 'divide', 'divmod', 'dot', 'drop', 'drop_duplicates', 'droplevel', 'dropna', 'dt', 'dtype', 'dtypes', 'duplicated', 'empty', 'eq', 'equals', 'ewm', 'expanding', 'explode', 'factorize', 'ffill', 'fillna', 'filter', 'first', 'first_valid_index', 'floordiv', 'ge', 'get', 'groupby', 'gt', 'hasnans', 'head', 'hist', 'iat', 'idxmax', 'idxmin', 'iloc', 'index', 'infer_objects', 'interpolate', 'is_monotonic', 'is_monotonic_decreasing', 'is_monotonic_increasing', 'is_unique', 'isin', 'isna', 'isnull', 'item', 'items', 'iteritems', 'keys', 'kurt', 'kurtosis', 'last', 'last_valid_index', 'le', 'loc', 'lt', 'mad', 'map', 'mask', 'max', 'mean', 'median', 'memory_usage', 'min', 'mod', 'mode', 'mul', 'multiply', 'name', 'nbytes', 'ndim', 'ne', 'nlargest', 'notna', 'notnull', 'nsmallest', 'nunique', 'pct_change', 'pipe', 'plot', 'pop', 'pow', 'prod', 'product', 'quantile', 'radd', 'rank', 'ravel', 'rdiv', 'rdivmod', 'reindex', 'reindex_like', 'rename', 'rename_axis', 'reorder_levels', 'repeat', 'replace', 'resample', 'reset_index', 'rfloordiv', 'rmod', 'rmul', 'rolling', 'round', 'rpow', 'rsub', 'rtruediv', 'sample', 'searchsorted', 'sem', 'set_axis', 'shape', 'shift', 'size', 'skew', 'slice_shift', 'sort_index', 'sort_values', 'sparse', 'squeeze', 'std', 'str', 'sub', 'subtract', 'sum', 'swapaxes', 'swaplevel', 'tail', 'take', 'to_clipboard', 'to_csv', 'to_dict', 'to_excel', 'to_frame', 'to_hdf', 'to_json', 'to_latex', 'to_list', 'to_markdown', 'to_numpy', 'to_period', 'to_pickle', 'to_sql', 'to_string', 'to_timestamp', 'to_xarray', 'tolist', 'transform', 'transpose', 'truediv', 'truncate', 'tshift', 'tz_convert', 'tz_localize', 'unique', 'unstack', 'update', 'value_counts', 'values', 'var', 'view', 'where', 'xs']
# Let's import scipy.stats library
import scipy.stats as stats
# The next topic for this tutorial is to explain and interpret p-values, correlation and covariance
# You probably have heard of these three terms but are struggled in figuring out their meaning and what
# context should we apply them.
# Let's look at the help documentation of stats.pearsonr first
#help(stats.pearsonr)
Help on function pearsonr in module scipy.stats.stats: pearsonr(x, y) Pearson correlation coefficient and p-value for testing non-correlation. The Pearson correlation coefficient [1]_ measures the linear relationship between two datasets. The calculation of the p-value relies on the assumption that each dataset is normally distributed. (See Kowalski [3]_ for a discussion of the effects of non-normality of the input on the distribution of the correlation coefficient.) Like other correlation coefficients, this one varies between -1 and +1 with 0 implying no correlation. Correlations of -1 or +1 imply an exact linear relationship. Positive correlations imply that as x increases, so does y. Negative correlations imply that as x increases, y decreases. The p-value roughly indicates the probability of an uncorrelated system producing datasets that have a Pearson correlation at least as extreme as the one computed from these datasets. Parameters ---------- x : (N,) array_like Input array. y : (N,) array_like Input array. Returns ------- r : float Pearson's correlation coefficient. p-value : float Two-tailed p-value. Warns ----- PearsonRConstantInputWarning Raised if an input is a constant array. The correlation coefficient is not defined in this case, so ``np.nan`` is returned. PearsonRNearConstantInputWarning Raised if an input is "nearly" constant. The array ``x`` is considered nearly constant if ``norm(x - mean(x)) < 1e-13 * abs(mean(x))``. Numerical errors in the calculation ``x - mean(x)`` in this case might result in an inaccurate calculation of r. See Also -------- spearmanr : Spearman rank-order correlation coefficient. kendalltau : Kendall's tau, a correlation measure for ordinal data. Notes ----- The correlation coefficient is calculated as follows: .. math:: r = \frac{\sum (x - m_x) (y - m_y)} {\sqrt{\sum (x - m_x)^2 \sum (y - m_y)^2}} where :math:`m_x` is the mean of the vector :math:`x` and :math:`m_y` is the mean of the vector :math:`y`. Under the assumption that x and y are drawn from independent normal distributions (so the population correlation coefficient is 0), the probability density function of the sample correlation coefficient r is ([1]_, [2]_):: (1 - r**2)**(n/2 - 2) f(r) = --------------------- B(1/2, n/2 - 1) where n is the number of samples, and B is the beta function. This is sometimes referred to as the exact distribution of r. This is the distribution that is used in `pearsonr` to compute the p-value. The distribution is a beta distribution on the interval [-1, 1], with equal shape parameters a = b = n/2 - 1. In terms of SciPy's implementation of the beta distribution, the distribution of r is:: dist = scipy.stats.beta(n/2 - 1, n/2 - 1, loc=-1, scale=2) The p-value returned by `pearsonr` is a two-sided p-value. For a given sample with correlation coefficient r, the p-value is the probability that abs(r') of a random sample x' and y' drawn from the population with zero correlation would be greater than or equal to abs(r). In terms of the object ``dist`` shown above, the p-value for a given r and length n can be computed as:: p = 2*dist.cdf(-abs(r)) When n is 2, the above continuous distribution is not well-defined. One can interpret the limit of the beta distribution as the shape parameters a and b approach a = b = 0 as a discrete distribution with equal probability masses at r = 1 and r = -1. More directly, one can observe that, given the data x = [x1, x2] and y = [y1, y2], and assuming x1 != x2 and y1 != y2, the only possible values for r are 1 and -1. Because abs(r') for any sample x' and y' with length 2 will be 1, the two-sided p-value for a sample of length 2 is always 1. References ---------- .. [1] "Pearson correlation coefficient", Wikipedia, https://en.wikipedia.org/wiki/Pearson_correlation_coefficient .. [2] Student, "Probable error of a correlation coefficient", Biometrika, Volume 6, Issue 2-3, 1 September 1908, pp. 302-310. .. [3] C. J. Kowalski, "On the Effects of Non-Normality on the Distribution of the Sample Product-Moment Correlation Coefficient" Journal of the Royal Statistical Society. Series C (Applied Statistics), Vol. 21, No. 1 (1972), pp. 1-12. Examples -------- >>> from scipy import stats >>> a = np.array([0, 0, 0, 1, 1, 1, 1]) >>> b = np.arange(7) >>> stats.pearsonr(a, b) (0.8660254037844386, 0.011724811003954649) >>> stats.pearsonr([1, 2, 3, 4, 5], [10, 9, 2.5, 6, 4]) (-0.7426106572325057, 0.1505558088534455)
The stats.pearsonr function takes two arrays as input and returns two float values, the Pearson's correlation and the p-value.
You can see in the documentation that correlation takes a value between -1 and +1, with 0 implying no correlation. We define
# .corr(), .cov()
# Let's revisit the dataset you encountered in the assignment 2
# For example, you may feel interested in understanding whether any correlation between
# mother's education and the likelihood of children to ever experience chicken pox
import pandas as pd
cdc_2017_data = pd.read_csv('NISPUF17.csv', index_col = 0)
cdc_2017_data.head()
| SEQNUMC | SEQNUMHH | PDAT | PROVWT_D | RDDWT_D | STRATUM | YEAR | AGECPOXR | HAD_CPOX | AGEGRP | ... | XVRCTY2 | XVRCTY3 | XVRCTY4 | XVRCTY5 | XVRCTY6 | XVRCTY7 | XVRCTY8 | XVRCTY9 | INS_STAT2_I | INS_BREAK_I | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 128521 | 12852 | 2 | NaN | 235.916956 | 1031 | 2017 | NaN | 2 | 1 | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ||
| 2 | 10741 | 1074 | 2 | NaN | 957.353840 | 1068 | 2017 | NaN | 2 | 1 | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ||
| 3 | 220011 | 22001 | 2 | NaN | 189.611299 | 1050 | 2017 | NaN | 2 | 3 | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ||
| 4 | 86131 | 8613 | 1 | 675.430817 | 333.447418 | 1040 | 2017 | NaN | 2 | 1 | ... | NaN | NaN | NaN | NaN | NaN | NaN | 1.0 | 2.0 | ||
| 5 | 227141 | 22714 | 1 | 482.617748 | 278.768063 | 1008 | 2017 | NaN | 2 | 1 | ... | NaN | NaN | NaN | NaN | NaN | NaN | 2.0 | 1.0 |
5 rows × 453 columns
# We would pause and go to the user's guide to find out the column that describes the eudcation
# level of children's mother, and the indicator of child ever have chicken pox
# EDUC1 - 1: < 12 years, 2: 12 year, 3: College, 4: College graduate
cdc_2017_data['EDUC1']
cdc_2017_data['EDUC1'].unique()
array([4, 3, 1, 2])
cdc_2017_data['HAD_CPOX'].unique()
array([ 2, 1, 77, 99])
# We filter only the observations where children either have or haven't got chicken pox
test_df = cdc_2017_data[(cdc_2017_data['HAD_CPOX'] ==1 ) | (cdc_2017_data['HAD_CPOX'] ==2 )][['EDUC1', 'HAD_CPOX']]
# and apply stats.pearsonr to return the correlation and p-value for columns in test_df
corr, pval = stats.pearsonr(test_df['EDUC1'], test_df['HAD_CPOX'])
# Let's print out the results
print("Correlation:", corr, ", p-value:", pval)
Correlation: 0.04061483261248254 , p-value: 7.816541295447035e-12
So you can see the correlation is 0.0406 something, and p-value is 7.8165 * e-12, a very very small value. Let's take a close look about the meaning of correlation and p-value.
Having higher education for parent is not correlated with the probability of a child to ever experience chicken pox
The alternative hypothesis, on the other hand, states that the two columns are correlated, meaning having higher education for parent is correlated with the probability of a child to ever experience chicken pox. But we need evidence to make a conclusion whether or not we are in favor of this hypothesis.
p-value is the probability that we observed a correlation between EDUC1 and HAD_CPOX to be as extreme as, or more extreme than the correlation we obtained through data (0.0406) by chance.
So the probability of observing a correlation between EDUC1 and HAD_CPOX to be at least as strong as 0.0406 by chance is very low (7.816541295447035e-12)
A nearly zero p-value means that it is highly unlikely to observe (at least such strong) correlation (between mother's education level and child ever experienced chicken pox by chance), assuming that the null hypothesis to be true.
To test the hypothesis, we will choose a significance level (\alpha) as threshold, where commonly used alpha values are 0.05, 0.01. We say that we have sufficient evidence to reject the null hypothesis and in favor of the alternative hypothesis if the p-value is lower than the specified value (alpha).
Since the p-value is less than the significance level, we conclude that
mother's education is a statistically significant factor associated with lowering the likelihood of children to ever experience chicken pox.
Both correlation and covariance measure the relationship and the dependency between two variables.
“Covariance” indicates the direction of the linear relationship between variables.
Positive covariance indicates the direction of the linear relationship between variables is positive, while negative covariance indicates the direction of the linear relationship between variables is negative.
“Correlation” measures both the strength and direction of the linear relationship between two variables. People would prefer using correlation if they want to examine how strong the linear relationship exhibits in the variables they are interested.
# So let's take a look about the grades dataset
# It shows the assignment score, tutorial score, midterm score, take home
# project score and final grade for each students in a class
grades = pd.read_csv('datasets/class_grades.csv')
grades.head()
| Prefix | Assignment | Tutorial | Midterm | TakeHome | Final | |
|---|---|---|---|---|---|---|
| 0 | 5 | 57.14 | 34.09 | 64.38 | 51.48 | 52.50 |
| 1 | 8 | 95.05 | 105.49 | 67.50 | 99.07 | 68.33 |
| 2 | 8 | 83.70 | 83.17 | NaN | 63.15 | 48.89 |
| 3 | 7 | NaN | NaN | 49.38 | 105.93 | 80.56 |
| 4 | 8 | 91.32 | 93.64 | 95.00 | 107.41 | 73.89 |
# We first examine the covariance by using .cov() function
# That allows you to see a matrix, in which every entry gives you how much two random
# variables vary together, or the join variability of every two-variable pairs
# You would not be surprised that the covariance matrix returned is always symmetric
# Meanwhile, you can see diagonal coveriance must be positive
grades.cov()
| Prefix | Assignment | Tutorial | Midterm | TakeHome | Final | |
|---|---|---|---|---|---|---|
| Prefix | 0.870336 | -0.218565 | 6.363605 | -0.709000 | -3.475591e+02 | 1.582513 |
| Assignment | -0.218565 | 160.397946 | 72.276580 | 59.374144 | 4.017545e+03 | 70.858213 |
| Tutorial | 6.363605 | 72.276580 | 231.766788 | 39.110860 | 6.581294e+03 | 67.203422 |
| Midterm | -0.709000 | 59.374144 | 39.110860 | 369.614271 | -1.036180e+03 | 265.144864 |
| TakeHome | -347.559076 | 4017.545275 | 6581.294121 | -1036.179801 | 6.852749e+06 | -3099.644143 |
| Final | 1.582513 | 70.858213 | 67.203422 | 265.144864 | -3.099644e+03 | 353.480882 |
# Let let's look at Correlation by using .corr() function
# The square of correlation r is actually R^2, which indicates the percent of variance explained in a dataset
# for the variables we included in a model.
# Correlation only works well with linear relationship (exponentialb)
grades.corr()
| Prefix | Assignment | Tutorial | Midterm | TakeHome | Final | |
|---|---|---|---|---|---|---|
| Prefix | 1.000000 | -0.018342 | 0.442884 | -0.039544 | -0.141028 | 0.090255 |
| Assignment | -0.018342 | 1.000000 | 0.392747 | 0.241723 | 0.120416 | 0.295852 |
| Tutorial | 0.442884 | 0.392747 | 1.000000 | 0.134117 | 0.159028 | 0.235087 |
| Midterm | -0.039544 | 0.241723 | 0.134117 | 1.000000 | -0.020095 | 0.723742 |
| TakeHome | -0.141028 | 0.120416 | 0.159028 | -0.020095 | 1.000000 | -0.061463 |
| Final | 0.090255 | 0.295852 | 0.235087 | 0.723742 | -0.061463 | 1.000000 |
Correlation 1 in diagonal elements mean there is perfect linear relationship for same column pairs
You would see for example, positive values for Assignment-Final, Tutorial-Final, Midterm-Final pairs and negative value for TakeHome-Final pair. What do these tell us?
The higher score students achieve in Assignment, Tutorial and Midterm, the higher students would likely to achieve in their final examination. That forms the positively correlated relationship between coursework and the Final exam score.
# It is interesting to examine whether the negative correlation between Take home score
# and the final exam score occurred by chance.
# We can use stats.pearsonr() function again to return the p-value
corr, p_value = stats.pearsonr(grades['TakeHome'], grades['Final'])
# So we got missing values in either column, let's get rid of that
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-157-8b08bcdfb7eb> in <module> 4 # We can use stats.pearsonr() function again to return the p-value 5 ----> 6 corr, p_value = stats.pearsonr(grades['TakeHome'], grades['Final']) /opt/anaconda3/lib/python3.8/site-packages/scipy/stats/stats.py in pearsonr(x, y) 3865 # scipy.linalg.norm(xm) does not overflow if xm is, for example, 3866 # [-5e210, 5e210, 3e200, -3e200] -> 3867 normxm = linalg.norm(xm) 3868 normym = linalg.norm(ym) 3869 /opt/anaconda3/lib/python3.8/site-packages/scipy/linalg/misc.py in norm(a, ord, axis, keepdims, check_finite) 138 # Differs from numpy only in non-finite handling and the use of blas. 139 if check_finite: --> 140 a = np.asarray_chkfinite(a) 141 else: 142 a = np.asarray(a) /opt/anaconda3/lib/python3.8/site-packages/numpy/lib/function_base.py in asarray_chkfinite(a, dtype, order) 496 a = asarray(a, dtype=dtype, order=order) 497 if a.dtype.char in typecodes['AllFloat'] and not np.isfinite(a).all(): --> 498 raise ValueError( 499 "array must not contain infs or NaNs") 500 return a ValueError: array must not contain infs or NaNs
# We use .isnull().sum() to check the number of NaN in student grades dataset
grades.isnull().sum()
Prefix 0 Assignment 7 Tutorial 3 Midterm 3 TakeHome 5 Final 3 Rank 0 dtype: int64
# Fill NaN with mean(), min(), max() of each column in student grades dataset
# apply() function in Pandas is one of the idiom that takes a function and take
# the axis on which to operate as parameters.
grades.apply(lambda x: x.fillna(x.mean()), axis = 0)
# Note that fillna method would not automatically overwrite the dataset
| Prefix | Assignment | Tutorial | Midterm | TakeHome | Final | Rank | |
|---|---|---|---|---|---|---|---|
| 68 | 7 | 100.83 | 105.57000 | 101.25 | 104.44 | 113.890000 | 1.0 |
| 33 | 7 | 98.58 | 91.03000 | 101.25 | 104.26 | 112.780000 | 2.0 |
| 75 | 7 | 93.83 | 89.60625 | 106.25 | 94.44 | 107.780000 | 3.0 |
| 48 | 8 | 95.90 | 99.99000 | 95.62 | 105.56 | 107.220000 | 4.0 |
| 63 | 7 | 95.00 | 94.36000 | 90.62 | 100.93 | 106.110000 | 5.0 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 22 | 8 | 63.78 | 76.21000 | 39.38 | 42.22 | 39.440000 | 95.0 |
| 56 | 8 | 87.93 | 106.53000 | 37.50 | 97.41 | 33.060000 | 96.0 |
| 60 | 8 | 71.79 | 102.87000 | 41.88 | 24.77 | 73.414375 | 99.0 |
| 20 | 8 | 63.40 | 86.21000 | 63.12 | 72.78 | 73.414375 | 99.0 |
| 38 | 6 | 90.74 | 89.64000 | 61.25 | 90.00 | 73.414375 | 99.0 |
99 rows × 7 columns
# Using the same logic we can impute NaN with median of each column too!
# x.median() with median and x.mode() with mode by replacing the function mean()
grades.apply(lambda x: x.fillna(x.median()), axis = 0)
grades = grades.apply(lambda x: x.fillna(x.median()), axis = 0)
# Now we imputed the NaN values with the median of their corresponding column.
# Let's now come back and see the p-value
corr, p_value = stats.pearsonr(grades['TakeHome'], grades['Final'])
# Let's print out the results
print("Correlation:", corr, ", p-value:", p_value)
Correlation: -0.06452030726571288 , p-value: 0.5257693151703908
Higher take home score is not correlated with the lower final exam score
It is a bit awkward at the first sight, but essentially we already see the negative correlation. So the null hypothesis is claiming that obtaining higher take home score is not associated with a lower final exam score.
The alternative hypothesis, on the other hand, states that the two columns are correlated, meaning having higher take home score is correlated with the lower final exam score. But we need evidence to make a conclusion whether or not we are in favor of this hypothesis.
p-value is the probability that we observed a correlation between TakeHome and Final that is as extreme as, or more extreme than the correlation we obtained through data (-0.0645) by chance.
So the probability of observing a correlation between TakeHome and Final to be at least as strong as -0.0645 by chance is very high (0.525769)
A p-value of around 0.5 means that it is likely to observe the correlation (or more stronger correlation) by chance, assuming that the null hypothesis to be true.
To test the hypothesis, we will choose a significance level (\alpha) as threshold, where commonly used alpha values are 0.05, 0.01. We say that we do not have sufficient evidence to reject the null hypothesis, since p-value is greater than the specified value (alpha).
Since the p-value is greater than the significance level, we conclude that
We do not have sufficient evidence to claim that higher take home score is correlated with the lower final exam score.
# Broadcasting
grades.Final +=5
# We didn't use the inplace option
# Overwrite the original dataset without explicitly making a copy of the original data
# ,but 5 points have been added to each student as extra credit
grades
| Prefix | Assignment | Tutorial | Midterm | TakeHome | Final | |
|---|---|---|---|---|---|---|
| 0 | 5 | 57.14 | 34.09 | 64.38 | 51.48 | 57.50 |
| 1 | 8 | 95.05 | 105.49 | 67.50 | 99.07 | 73.33 |
| 2 | 8 | 83.70 | 83.17 | NaN | 63.15 | 53.89 |
| 3 | 7 | NaN | NaN | 49.38 | 105.93 | 85.56 |
| 4 | 8 | 91.32 | 93.64 | 95.00 | 107.41 | 78.89 |
| ... | ... | ... | ... | ... | ... | ... |
| 94 | 8 | NaN | 103.71 | 45.00 | 93.52 | 66.94 |
| 95 | 7 | NaN | 80.54 | 41.25 | 93.70 | 44.72 |
| 96 | 8 | 89.94 | 102.77 | 87.50 | 90.74 | 92.78 |
| 97 | 7 | 95.60 | 76.13 | 66.25 | 99.81 | 90.56 |
| 98 | 8 | 63.40 | 97.37 | 73.12 | 72.78 | 82.22 |
99 rows × 6 columns
# Return a boolean true if student's final grade is greater than 50
grades.Final.gt(50)
0 True 1 True 2 False 3 True 4 True ... 94 True 95 False 96 True 97 True 98 True Name: Final, Length: 99, dtype: bool
# Return a boolean true if student's final grade is less than 70
grades.Final.lt(70)
0 True 1 True 2 True 3 False 4 False ... 94 True 95 True 96 False 97 False 98 False Name: Final, Length: 99, dtype: bool
# Filter the rows that students' final score is greater than 50
grades[grades.Final.gt(50)]
| Prefix | Assignment | Tutorial | Midterm | TakeHome | Final | |
|---|---|---|---|---|---|---|
| 0 | 5 | 57.14 | 34.09 | 64.38 | 51.48 | 52.50 |
| 1 | 8 | 95.05 | 105.49 | 67.50 | 99.07 | 68.33 |
| 3 | 7 | NaN | NaN | 49.38 | 105.93 | 80.56 |
| 4 | 8 | 91.32 | 93.64 | 95.00 | 107.41 | 73.89 |
| 5 | 7 | 95.00 | 92.58 | 93.12 | 97.78 | 68.06 |
| ... | ... | ... | ... | ... | ... | ... |
| 93 | 8 | 87.52 | 91.58 | 56.25 | 71.85 | 85.00 |
| 94 | 8 | NaN | 103.71 | 45.00 | 93.52 | 61.94 |
| 96 | 8 | 89.94 | 102.77 | 87.50 | 90.74 | 87.78 |
| 97 | 7 | 95.60 | 76.13 | 66.25 | 99.81 | 85.56 |
| 98 | 8 | 63.40 | 97.37 | 73.12 | 72.78 | 77.22 |
76 rows × 6 columns
cdc_2017_data.head()
| SEQNUMC | SEQNUMHH | PDAT | PROVWT_D | RDDWT_D | STRATUM | YEAR | AGECPOXR | HAD_CPOX | AGEGRP | ... | XVRCTY2 | XVRCTY3 | XVRCTY4 | XVRCTY5 | XVRCTY6 | XVRCTY7 | XVRCTY8 | XVRCTY9 | INS_STAT2_I | INS_BREAK_I | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 128521 | 12852 | 2 | NaN | 235.916956 | 1031 | 2017 | NaN | 2 | 1 | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ||
| 2 | 10741 | 1074 | 2 | NaN | 957.353840 | 1068 | 2017 | NaN | 2 | 1 | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ||
| 3 | 220011 | 22001 | 2 | NaN | 189.611299 | 1050 | 2017 | NaN | 2 | 3 | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ||
| 4 | 86131 | 8613 | 1 | 675.430817 | 333.447418 | 1040 | 2017 | NaN | 2 | 1 | ... | NaN | NaN | NaN | NaN | NaN | NaN | 1.0 | 2.0 | ||
| 5 | 227141 | 22714 | 1 | 482.617748 | 278.768063 | 1008 | 2017 | NaN | 2 | 1 | ... | NaN | NaN | NaN | NaN | NaN | NaN | 2.0 | 1.0 |
5 rows × 453 columns
# How many children without taking varicella vaccine have never experienced chicken pox?
len(cdc_2017_data[(cdc_2017_data['HAD_CPOX'] == 2) & (cdc_2017_data['P_NUMVRC'] == 0.0)][['HAD_CPOX', 'P_NUMVRC']])
1283
# How many children who took at least one varicella vaccine have never experienced chicken pox?
len(cdc_2017_data[(cdc_2017_data['HAD_CPOX'] == 2) & (cdc_2017_data['P_NUMVRC'] > 0.0)][['HAD_CPOX', 'P_NUMVRC']])
13830
# How many children without taking varicella vaccine experienced chicken pox in their childhood?
len(cdc_2017_data[(cdc_2017_data['HAD_CPOX'] == 1) & (cdc_2017_data['P_NUMVRC'] == 0.0)][['HAD_CPOX', 'P_NUMVRC']])
52
# How many children who took at least one varicella vaccine experienced chicken pox in their childhood?
len(cdc_2017_data[(cdc_2017_data['HAD_CPOX'] == 1) & (cdc_2017_data['P_NUMVRC'] > 0.0)][['HAD_CPOX', 'P_NUMVRC']])
121
# Question: What is the proportion of children ever experienced chicken pox
# if he/she did not take varicella vaccine?
print(52/ (1283 + 52))
# Question: What is the proportion of children ever experienced chicken pox
# if he/she took at least one varicella vaccine?
print(121/ (121 + 13830))
0.03895131086142322 0.008673213389721168
# Return the fourth row
print(cdc_2017_data.iloc[3])
# Return the element located at the fourth row, fourth column
print(cdc_2017_data.iloc[3, 3])
SEQNUMC 86131 SEQNUMHH 8613 PDAT 1 PROVWT_D 675.431 RDDWT_D 333.447 ... XVRCTY7 NaN XVRCTY8 NaN XVRCTY9 NaN INS_STAT2_I 1 INS_BREAK_I 2 Name: 4, Length: 453, dtype: object<PHONE_NUMBER>14971
# You need to specify index, or otherwise it is common to return you a KeyError
#cdc_2017_data.loc['0']
# using TAB after typing % will show you a grand list of available inline magics, Error Types
%
grades
| Prefix | Assignment | Tutorial | Midterm | TakeHome | Final | Rank | |
|---|---|---|---|---|---|---|---|
| 68 | 7 | 100.83 | 105.570 | 101.25 | 104.44 | 113.89 | 1.0 |
| 33 | 7 | 98.58 | 91.030 | 101.25 | 104.26 | 112.78 | 2.0 |
| 75 | 7 | 93.83 | 93.015 | 106.25 | 94.44 | 107.78 | 3.0 |
| 48 | 8 | 95.90 | 99.990 | 95.62 | 105.56 | 107.22 | 4.0 |
| 63 | 7 | 95.00 | 94.360 | 90.62 | 100.93 | 106.11 | 5.0 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 22 | 8 | 63.78 | 76.210 | 39.38 | 42.22 | 39.44 | 95.0 |
| 56 | 8 | 87.93 | 106.530 | 37.50 | 97.41 | 33.06 | 96.0 |
| 60 | 8 | 71.79 | 102.870 | 41.88 | 24.77 | 71.25 | 99.0 |
| 20 | 8 | 63.40 | 86.210 | 63.12 | 72.78 | 71.25 | 99.0 |
| 38 | 6 | 90.74 | 89.640 | 61.25 | 90.00 | 71.25 | 99.0 |
99 rows × 7 columns
# What do you think the rank function do?
# By default, the rank with sort the values based on ascending order
# For ties, by default the rank function will average the different rank numbers (2 and 3)
# can you pass a custom method for splitting ties ?
# How to reverse the rank
grades['Rank'] = grades['Final'].rank(method = 'max', na_option = 'bottom', ascending = False)
grades.sort_values("Rank", inplace = True)
grades
| Prefix | Assignment | Tutorial | Midterm | TakeHome | Final | Rank | |
|---|---|---|---|---|---|---|---|
| 68 | 7 | 100.83 | 105.57 | 101.25 | 104.44 | 113.89 | 1.0 |
| 33 | 7 | 98.58 | 91.03 | 101.25 | 104.26 | 112.78 | 2.0 |
| 75 | 7 | 93.83 | NaN | 106.25 | 94.44 | 107.78 | 3.0 |
| 48 | 8 | 95.90 | 99.99 | 95.62 | 105.56 | 107.22 | 4.0 |
| 63 | 7 | 95.00 | 94.36 | 90.62 | 100.93 | 106.11 | 5.0 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 22 | 8 | 63.78 | 76.21 | 39.38 | 42.22 | 39.44 | 95.0 |
| 56 | 8 | 87.93 | 106.53 | 37.50 | 97.41 | 33.06 | 96.0 |
| 60 | 8 | 71.79 | 102.87 | 41.88 | 24.77 | NaN | 99.0 |
| 20 | 8 | 63.40 | 86.21 | 63.12 | 72.78 | NaN | 99.0 |
| 38 | 6 | 90.74 | 89.64 | 61.25 | 90.00 | NaN | 99.0 |
99 rows × 7 columns
Next time we will talk about more about Pandas. Topics are, but not limited to:
# Mutate a column .mutate %>%
# _pandorable pandas code


Adam Zyglis: The silencer 19 Nov 2019, 18:16 by https://www.facebook.com/zyglis Click here to see more of Pulitzer Prize-winner Adam Zyglis' political cartoons.