Importing demultiplexed sequence data

Importing demultiplexed sequence data

In this section of the tutorial, we’ll import raw fastq data that is already demultiplexed (i.e., separated into per-sample fastq files) into a QIIME 2 artifact.

Importing

We’ll begin with the data import.

Using the Upload Data tool:
  • Steps to setup data_to_import:sequences:

    1. On the fourth tab (Rule-based):

      1. Set “Upload data as” to Collection(s)

      2. Set “Load tabular data from” to Pasted Table

      3. Paste the following contents into the large text area:

        FMT.0093C_46_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093C_46_L001_R2_001.fastq.gz
        FMT.0093C_5_L001_R1_001.fastq.gz    https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093C_5_L001_R1_001.fastq.gz
        FMT.0093D_2_L001_R1_001.fastq.gz    https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093D_2_L001_R1_001.fastq.gz
        FMT.0093D_43_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093D_43_L001_R2_001.fastq.gz
        FMT.0093E_25_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093E_25_L001_R1_001.fastq.gz
        FMT.0093E_66_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093E_66_L001_R2_001.fastq.gz
        FMT.0093F_47_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093F_47_L001_R2_001.fastq.gz
        FMT.0093F_6_L001_R1_001.fastq.gz    https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093F_6_L001_R1_001.fastq.gz
        FMT.0093G_24_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093G_24_L001_R1_001.fastq.gz
        FMT.0093G_65_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093G_65_L001_R2_001.fastq.gz
        FMT.0093H_26_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093H_26_L001_R1_001.fastq.gz
        FMT.0093H_67_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093H_67_L001_R2_001.fastq.gz
        FMT.0093I_1_L001_R1_001.fastq.gz    https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093I_1_L001_R1_001.fastq.gz
        FMT.0093I_42_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093I_42_L001_R2_001.fastq.gz
        FMT.0093J_22_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093J_22_L001_R1_001.fastq.gz
        FMT.0093J_63_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093J_63_L001_R2_001.fastq.gz
        FMT.0093K_50_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093K_50_L001_R2_001.fastq.gz
        FMT.0093K_9_L001_R1_001.fastq.gz    https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093K_9_L001_R1_001.fastq.gz
        FMT.0093L_3_L001_R1_001.fastq.gz    https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093L_3_L001_R1_001.fastq.gz
        FMT.0093L_44_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093L_44_L001_R2_001.fastq.gz
        FMT.0093M_28_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093M_28_L001_R1_001.fastq.gz
        FMT.0093M_69_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093M_69_L001_R2_001.fastq.gz
        FMT.0093P_14_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093P_14_L001_R1_001.fastq.gz
        FMT.0093P_55_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093P_55_L001_R2_001.fastq.gz
        FMT.0093Q_39_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093Q_39_L001_R1_001.fastq.gz
        FMT.0093Q_80_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093Q_80_L001_R2_001.fastq.gz
        FMT.0093S_30_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093S_30_L001_R1_001.fastq.gz
        FMT.0093S_71_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093S_71_L001_R2_001.fastq.gz
        FMT.0093T_35_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093T_35_L001_R1_001.fastq.gz
        FMT.0093T_76_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093T_76_L001_R2_001.fastq.gz
        FMT.0093U_13_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093U_13_L001_R1_001.fastq.gz
        FMT.0093U_54_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093U_54_L001_R2_001.fastq.gz
        FMT.0093V_33_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093V_33_L001_R1_001.fastq.gz
        FMT.0093V_74_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093V_74_L001_R2_001.fastq.gz
        FMT.0093W_18_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093W_18_L001_R1_001.fastq.gz
        FMT.0093W_59_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093W_59_L001_R2_001.fastq.gz
        FMT.0093X_11_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093X_11_L001_R1_001.fastq.gz
        FMT.0093X_52_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0093X_52_L001_R2_001.fastq.gz
        FMT.0103V_27_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0103V_27_L001_R1_001.fastq.gz
        FMT.0103V_68_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0103V_68_L001_R2_001.fastq.gz
        FMT.0103W_0_L001_R1_001.fastq.gz    https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0103W_0_L001_R1_001.fastq.gz
        FMT.0103W_41_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0103W_41_L001_R2_001.fastq.gz
        FMT.0106H_49_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106H_49_L001_R2_001.fastq.gz
        FMT.0106H_8_L001_R1_001.fastq.gz    https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106H_8_L001_R1_001.fastq.gz
        FMT.0106I_23_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106I_23_L001_R1_001.fastq.gz
        FMT.0106I_64_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106I_64_L001_R2_001.fastq.gz
        FMT.0106L_21_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106L_21_L001_R1_001.fastq.gz
        FMT.0106L_62_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106L_62_L001_R2_001.fastq.gz
        FMT.0106M_45_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106M_45_L001_R2_001.fastq.gz
        FMT.0106M_4_L001_R1_001.fastq.gz    https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106M_4_L001_R1_001.fastq.gz
        FMT.0106N_29_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106N_29_L001_R1_001.fastq.gz
        FMT.0106N_70_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106N_70_L001_R2_001.fastq.gz
        FMT.0106R_38_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106R_38_L001_R1_001.fastq.gz
        FMT.0106R_79_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0106R_79_L001_R2_001.fastq.gz
        FMT.0107B_15_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107B_15_L001_R1_001.fastq.gz
        FMT.0107B_56_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107B_56_L001_R2_001.fastq.gz
        FMT.0107C_40_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107C_40_L001_R1_001.fastq.gz
        FMT.0107C_81_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107C_81_L001_R2_001.fastq.gz
        FMT.0107D_32_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107D_32_L001_R1_001.fastq.gz
        FMT.0107D_73_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107D_73_L001_R2_001.fastq.gz
        FMT.0107E_17_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107E_17_L001_R1_001.fastq.gz
        FMT.0107E_58_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107E_58_L001_R2_001.fastq.gz
        FMT.0107F_34_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107F_34_L001_R1_001.fastq.gz
        FMT.0107F_75_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107F_75_L001_R2_001.fastq.gz
        FMT.0107G_12_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107G_12_L001_R1_001.fastq.gz
        FMT.0107G_53_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107G_53_L001_R2_001.fastq.gz
        FMT.0107H_19_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107H_19_L001_R1_001.fastq.gz
        FMT.0107H_60_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107H_60_L001_R2_001.fastq.gz
        FMT.0107J_10_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107J_10_L001_R1_001.fastq.gz
        FMT.0107J_51_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107J_51_L001_R2_001.fastq.gz
        FMT.0107K_36_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107K_36_L001_R1_001.fastq.gz
        FMT.0107K_77_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107K_77_L001_R2_001.fastq.gz
        FMT.0107L_31_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107L_31_L001_R1_001.fastq.gz
        FMT.0107L_72_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107L_72_L001_R2_001.fastq.gz
        FMT.0107M_16_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107M_16_L001_R1_001.fastq.gz
        FMT.0107M_57_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107M_57_L001_R2_001.fastq.gz
        FMT.0107N_37_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107N_37_L001_R1_001.fastq.gz
        FMT.0107N_78_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107N_78_L001_R2_001.fastq.gz
        FMT.0107P_20_L001_R1_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107P_20_L001_R1_001.fastq.gz
        FMT.0107P_61_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107P_61_L001_R2_001.fastq.gz
        FMT.0107T_48_L001_R2_001.fastq.gz   https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107T_48_L001_R2_001.fastq.gz
        FMT.0107T_7_L001_R1_001.fastq.gz    https://qiime2-workshops.s3.us-west-2.amazonaws.com/faes-jan2022/data/020-tutorial-upstream/030-importing/data_to_import/FMT.0107T_7_L001_R1_001.fastq.gz
        
      4. Press the build button at the bottom.

    2. In the resulting UI, do the following:

      1. Add a rule by pressing the + Rules button and choosing Add / Modify Column Definitions.

      2. In the sidebar:

        1. Press +Add Definition and select List Identifier(s), then select column A.

        2. Press +Add Definition and select URL.

        3. Change the dropdown above the button to be B. (You should see the table headers list A (List Identifier) and B (URL).)

        4. Press the Apply button.

    3. In the bottom right, set “Name” to be data_to_import:sequences

    4. Press the Upload button at the bottom right.

Using the qiime2 tools import tool:
  1. Set “Type of data to import” to SampleData[PairedEndSequencesWithQuality]

  2. Set “QIIME 2 file format to import from” to Casava One Eight Single Lane Per Sample Directory Format

  3. For import_sequences, do the following:

    1. Leave “Select a mechanism” as Use collection to import

    2. Set “elements” to #: data_to_import:sequences

    3. Leave “Append an extension?” as No.

  4. Press the Execute button.

Once completed, for the new entry in your history, use the Edit button to set the name as follows:

(Renaming is optional, but it will make any subsequent steps easier to complete.)

History Name

“Name” to set (be sure to press Save)

#: qiime2 tools import [...]

demultiplexed-sequences.qza

Generating and viewing a summary of the imported data

After the import is complete, you can generate a summary of the imported artifact. This summary contains several important pieces of information.

First, it tells you how many sequences were obtained for each of the samples. The expected number of sequences per sample will vary depending on the sequencing technology that was applied and the the number of samples that were multiplexed in your run. You should review this, and ensure that you are getting the expected number of sequences on average.

Second, this summary provides interactive figures that illustrate sequence quality. This will give you an overview of the quality of your sequencing run, and you’ll need to extract information from these plots to perform quality control on the data in the next step of the tutorial.

Using the qiime2 demux summarize tool:
  1. Set “data” to #: demultiplexed-sequences.qza

  2. Press the Execute button.

Once completed, for the new entry in your history, use the Edit button to set the name as follows:

(Renaming is optional, but it will make any subsequent steps easier to complete.)

History Name

“Name” to set (be sure to press Save)

#: qiime2 demux summarize [...] : visualization.qzv

demultiplexed-sequences-summ.qzv