Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 7.364591548s FAIL (2025-10-08 17:26:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 7.364625914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 13.882354932s FAIL (2025-10-08 17:25:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 13.882388125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 18.573607347s FAIL (2025-10-08 17:25:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 18.573642904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 9.553791257s FAIL (2025-10-08 17:23:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 9.553823148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 12.147054203s FAIL (2025-10-08 17:23:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 12.147088718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 23.30294532s FAIL (2025-10-08 17:22:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 23.302984455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.865416053s FAIL (2025-10-08 17:21:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 6.8654514s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 9.767459375s FAIL (2025-10-08 17:21:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 9.767494743s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.................................X.........................] duration: 8.23076166s FAIL (2025-10-08 17:19:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 8.230816914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.................................X........................] duration: 9.226292436s FAIL (2025-10-08 17:18:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 9.226326472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.................................X.......................] duration: 6.301910497s FAIL (2025-10-08 17:18:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 6.301937767s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.................................X......................] duration: 6.376118933s FAIL (2025-10-08 17:17:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 6.376218331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.................................X.....................] duration: 7.167754587s FAIL (2025-10-08 17:16:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 7.167784473s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.................................X....................] duration: 17.109493432s FAIL (2025-10-08 17:16:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 17.109528069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.................................X...................] duration: 16.793186181s FAIL (2025-10-08 17:15:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 16.793229153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.................................X..................] duration: 20.021065393s FAIL (2025-10-08 17:14:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 20.021110438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.................................X.................] duration: 25.309090965s FAIL (2025-10-08 17:13:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 25.309123517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.................................X................] duration: 6.253889412s FAIL (2025-10-08 17:12:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 6.253932574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.................................X...............] duration: 17.480538811s FAIL (2025-10-08 17:11:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 17.480577326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.................................X..............] duration: 6.600780323s FAIL (2025-10-08 17:11:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 6.600806824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.................................X.............] duration: 8.335197787s FAIL (2025-10-08 17:10:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 8.335229166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.................................X............] duration: 7.645708366s FAIL (2025-10-08 17:09:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 7.645738604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.................................X...........] duration: 7.776802278s FAIL (2025-10-08 17:08:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 7.776839268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.................................X..........] duration: 13.056799772s FAIL (2025-10-08 17:07:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 13.056848706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.................................X.........] duration: 22.369718097s FAIL (2025-10-08 17:06:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 22.369751562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.................................X........] duration: 7.435530065s FAIL (2025-10-08 17:05:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 7.43556417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.................................X.......] duration: 25.968322731s FAIL (2025-10-08 17:04:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 25.968397333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.................................X......] duration: 23.072438585s FAIL (2025-10-08 17:03:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 23.072469735s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.................................X.....] duration: 18.092003648s FAIL (2025-10-08 17:03:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 18.092056328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.................................X....] duration: 20.146564856s FAIL (2025-10-08 17:02:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 20.146591566s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.................................X...] duration: 8.313209548s FAIL (2025-10-08 17:01:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 8.313239224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.................................X..] duration: 8.970727971s FAIL (2025-10-08 17:01:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 8.970759861s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.................................X.] duration: 14.824113761s FAIL (2025-10-08 17:00:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 14.824136355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.................................X] duration: 30.003454192s FAIL (2025-10-08 17:00:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ) context deadline exceeded ======LOGS====== ments... Oct 8 17:00:03.564851754 (363h28m49.026944237s) [upload=319738]: Got next segment. Inline: false Oct 8 17:00:03.564896009 (363h28m49.026988492s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 17:00:03.564902922 (363h28m49.026995405s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 17:00:03.844824719 (363h28m49.306917222s) [seg_pos={0 0}, upload=319738]: Begin upload segment... Oct 8 17:00:03.844898439 (363h28m49.306990932s) [seg_pos={0 0}, upload=319738]: Done begin upload segment. Oct 8 17:00:03.844943616 (363h28m49.307036099s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.849883933 (363h28m49.311976427s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.850879740 (363h28m49.312972223s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.851567389 (363h28m49.313659872s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852262031 (363h28m49.314354514s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.852911218 (363h28m49.315003701s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854266449 (363h28m49.316358932s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.854908841 (363h28m49.317001324s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.855514495 (363h28m49.317606978s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859056480 (363h28m49.321148963s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.859820414 (363h28m49.321912897s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.860799629 (363h28m49.322892112s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.866270156 (363h28m49.328362639s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.867174027 (363h28m49.32926652s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.877859184 (363h28m49.339951668s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:03.879110537 (363h28m49.34120303s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Uploading piece... Oct 8 17:00:05.645304395 (363h28m51.107396878s) [upload=319738]: Next returned nil segment Oct 8 17:00:05.646049223 (363h28m51.108141706s) [upload=319738]: Waiting for error group managing segments... Oct 8 17:00:06.093846932 (363h28m51.555939415s) [noise=true , address=45.91.116.12:4226 , piece_id=UTDWYBDEOCWG5MPYYUY64NM62ZXNMVWPF2WYXM7JSLI4KCIRGUZQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.161454882 (363h28m51.623547366s) [noise=true , address=45.91.116.14:4190 , piece_id=YQBGCB5XD4453LJCTQQWFGKR76BTQF474Y3AYBYDZAQ66AVOQNCA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.174265036 (363h28m51.636357529s) [noise=true , address=45.91.116.11:4119 , piece_id=ENVN6QFFXJYXTZ7VPL7JWSF6DY6VOJZQI6BZWTUGLQMOXR6ISIGQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.213406208 (363h28m51.675498691s) [noise=true , address=45.91.116.11:4117 , piece_id=EYILAJGT4YJJZ2AL3URO7O2K7VG7A5WJP46RVG5AWPYSKJQHPMXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.227444601 (363h28m51.689537084s) [noise=true , address=77.90.31.18:4806 , piece_id=LIICTJLR2VB62IG3PJ3ULTDT3XKR5EKPM5ANKCP4HEVWEE54X4UA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.264790865 (363h28m51.726883349s) [noise=true , address=77.90.31.12:4592 , piece_id=7CE7NUXJX6QABFTQYMKO7LH6VNTCVZZAQGWIVGLKVDAZ7GMZW5BQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.290928727 (363h28m51.75302123s) [noise=true , address=77.90.31.12:4589 , piece_id=ASTNZGMPGOQFANH3D65BHCM57JB7SE4R66BNB7KNVQYTBVEQEB2Q, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.291921888 (363h28m51.754014401s) [noise=true , address=45.91.116.8:4047 , piece_id=TI3O3XDL5FSG7P4WJLXDHLX63ECEKQFWDXQCBGCN2SPIDIB7MIHQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.293095593 (363h28m51.755188077s) [noise=true , address=45.91.116.6:4975 , piece_id=SU3XPVZPZRLM2LKLGAZ3NRQPPLNFSPNVFKGZYXLIFQET5TBGBVXA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.328444284 (363h28m51.790536767s) [noise=true , address=45.91.116.7:4010 , piece_id=H6JTRWS5FJS3UZ4ZYB4L6BCTBXPN6T3MBGBQBM6KKFE7SX7RVJ2A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.330051514 (363h28m51.792143997s) [noise=true , address=77.90.31.13:4627 , piece_id=UXFNDROV5TQOVETK6JVAVKUCNBTCOL52UEGXL3WHRBEKGRG3PSPA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.410068384 (363h28m51.872160867s) [noise=true , address=145.79.227.6:4258 , piece_id=DXJYL26QL2B3ZDQ2EDVYYZMMBUHNKOBGUP5QTZBGF5P2BTPUZMRQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562214702 (363h28m52.024307185s) [noise=true , address=77.90.31.16:4735 , piece_id=A5Y3ADCEMO44M7CYSQXBGB6N46AOKDHKTDZVJBF22FIKQUALJZWQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: <nil> Oct 8 17:00:06.562272422 (363h28m52.024364905s) [seg_pos={0 0}, upload=319738]: Segment reached optimal threshold of 13 pieces. Oct 8 17:00:06.562464108 (363h28m52.024556591s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 8 17:00:06.562479997 (363h28m52.02457251s) [noise=true , address=145.79.227.12:4767 , piece_id=652EGUQQJGCORJ7KQHU4HELPPGPPBY4HYRJRZPUXE4EYVVDGNO5A, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562641185 (363h28m52.024733668s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54234->145.79.227.15:4876: use of closed network connection Oct 8 17:00:06.562655281 (363h28m52.024747764s) [noise=true , address=145.79.227.15:4876 , piece_id=UBCVBIWYKXBQCXSAXGGRWW22DH3CIU67RAV2JVUNT5NLEPOOYIUQ, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562772354 (363h28m52.024864847s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.7:4009: use of closed network connection Oct 8 17:00:06.562785189 (363h28m52.024877672s) [noise=true , address=45.91.116.7:4009 , piece_id=VHNTJQAILF6OCNKX6INZR7IHVG4MWRUJ4HXR2IBLPKEYLGXR6GYA, seg_pos={0 0}, upload=319738]: Long tail cancellation due to optimal threshold Oct 8 17:00:06.562813853 (363h28m52.024906337s) [seg_pos={0 0}, upload=319738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 17:00:06.562955142 (363h28m52.025047626s) [upload=319738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 17:00:06.562964841 (363h28m52.025057324s) [upload=319738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 17:00:06.897245087 (363h28m52.35933756s) [upload=319738]: Done uploading segments... Oct 8 17:00:06.897274163 (363h28m52.359366646s) [upload=319738]: Upload finished. err: <nil> Duration: 30.003487606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 15.353463381s FAIL (2025-10-08 16:59:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 15.35348922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 28.945391887s FAIL (2025-10-08 16:59:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 28.945418327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 15.311232032s FAIL (2025-10-08 16:58:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 15.311262931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 8.005572083s FAIL (2025-10-08 16:57:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 8.005608763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 11.227647376s FAIL (2025-10-08 16:56:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 11.227705677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 15.245459281s FAIL (2025-10-08 16:56:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 15.245511561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 11.328739262s FAIL (2025-10-08 16:55:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 11.328770532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 20.355328607s FAIL (2025-10-08 16:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 20.355361451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 10.256538058s FAIL (2025-10-08 16:53:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 10.256570261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.275310046s FAIL (2025-10-08 16:52:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 7.27533903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 12.114628909s FAIL (2025-10-08 16:52:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 12.114701818s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 25.167208887s FAIL (2025-10-08 16:52:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 25.167238634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 7.415658819s FAIL (2025-10-08 16:51:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 7.415700697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 13.429576916s FAIL (2025-10-08 16:50:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 13.429608637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 10.469924507s FAIL (2025-10-08 16:49:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 10.469947861s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 18.861754461s FAIL (2025-10-08 16:49:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ) context deadline exceeded ======LOGS====== Oct 8 16:33:40.035386299 (363h2m25.497478792s) [upload=319361]: Starting upload Oct 8 16:33:40.035490698 (363h2m25.497583171s) [upload=319361]: Done starting upload Oct 8 16:33:40.035601429 (363h2m25.497693912s) [upload=319361]: Uploading segments... Oct 8 16:33:40.035763989 (363h2m25.497856472s) [upload=319361]: Got next segment. Inline: false Oct 8 16:33:40.035962656 (363h2m25.498055139s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 16:33:40.035968528 (363h2m25.498061011s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 16:33:40.294365360 (363h2m25.756457833s) [seg_pos={0 0}, upload=319361]: Begin upload segment... Oct 8 16:33:40.294440523 (363h2m25.756533006s) [seg_pos={0 0}, upload=319361]: Done begin upload segment. Oct 8 16:33:40.294473827 (363h2m25.75656636s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.294558568 (363h2m25.756651051s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295037029 (363h2m25.757129512s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295429676 (363h2m25.75752216s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.295970086 (363h2m25.758062579s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296369676 (363h2m25.758462159s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296663677 (363h2m25.75875615s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.296945453 (363h2m25.759037936s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.297277445 (363h2m25.759369928s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.298503189 (363h2m25.760595672s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.299658179 (363h2m25.761750662s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.301302299 (363h2m25.763394782s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.302928536 (363h2m25.765021019s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.304245303 (363h2m25.766337786s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.305625151 (363h2m25.767717624s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:40.306624113 (363h2m25.768716596s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Uploading piece... Oct 8 16:33:42.005690413 (363h2m27.467782896s) [upload=319361]: Next returned nil segment Oct 8 16:33:42.005717735 (363h2m27.467810218s) [upload=319361]: Waiting for error group managing segments... Oct 8 16:33:42.281097745 (363h2m27.743190228s) [noise=true , address=145.79.227.14:4839 , piece_id=XEXSNJVKLG6KYXXEJY4EGNIFYENYIQIIP2XMGBJTR77ZTDPGHHAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294438249 (363h2m27.756531373s) [noise=true , address=145.79.227.7:4293 , piece_id=AWIINRE6O7BMNJDQOAMVUMOMWT7AITYOHURJJMLB4EJ2GCUO5GAA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.294490077 (363h2m27.75658273s) [noise=true , address=145.79.227.17:4969 , piece_id=TOSG3M6CKXDDDE4EHXITFSIV54GKYHD4FMAUXEWDRQ3TY2SBZTAQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.313116977 (363h2m27.77520945s) [noise=true , address=145.79.227.17:4971 , piece_id=JHDKLHNL67WPVYNTSCUKGYL6TBUYMCWFU5KJTL3YU5ZDCRO5H5GQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.327062161 (363h2m27.789154634s) [noise=true , address=145.79.227.13:4801 , piece_id=AQNED2RX2MRUI667FL4VHPPEADA2YZPPIHJCGQ42FBPMFW27QE5Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.338401614 (363h2m27.800494097s) [noise=true , address=145.79.227.4:4185 , piece_id=C2XD4FX42TXWZ2L4BEYBV64LFWL4Y53QTQGII62AWUJWAWMCTMZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.521166054 (363h2m27.983258538s) [noise=true , address=45.91.116.9:4262 , piece_id=7IHINWYXWNHIYVOXNKWQBIZTUWQDPUSIDU5JTZVDNKYQUQOYBQWA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.544962328 (363h2m28.007054811s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTODDDNIZMOT2WK2TQEOIJNPT62UD3W7LWLIWJMDH266EGSH3WZA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.561084858 (363h2m28.023177331s) [noise=true , address=77.90.31.15:4697 , piece_id=FPHLFAPWAMPJSZSYRLAQ4BQ76X6M25C6RWXECJHAYLLI7E6MAPWQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.562561851 (363h2m28.024654334s) [noise=true , address=77.90.31.10:4518 , piece_id=RZQ7BT7FVRZD4ZYS3GNKGCEZWLMMFE2N76OPJYFYPNIXNNSKM5YQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.567785227 (363h2m28.02987771s) [noise=true , address=77.90.31.13:4627 , piece_id=YXVNME2V7R3MYTSPAFZUL6YJKJ4T5YB4PISV7LS3II7ZOAHPTG3Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.573511081 (363h2m28.035603564s) [noise=true , address=45.91.116.12:4227 , piece_id=K7L2ZC34LNNK5GPEQO2I66BZMEIR77BT5SGIMOK5LKQY5WII2N4A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625255793 (363h2m28.087348266s) [noise=true , address=77.90.31.18:4805 , piece_id=QZAK6X532MKBZQ2YDKQZEGDGCFCMNMRBKE57HB7XN4QJ2G7RPLYQ, seg_pos={0 0}, upload=319361]: Done uploading piece. err: <nil> Oct 8 16:33:42.625285750 (363h2m28.087378233s) [seg_pos={0 0}, upload=319361]: Segment reached optimal threshold of 13 pieces. Oct 8 16:33:42.625412682 (363h2m28.087505155s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625419024 (363h2m28.087511507s) [noise=true , address=77.90.31.21:4877 , piece_id=EQUAQBZAMIFYU6UDQJTOZ2M7HKIEZ34L4KPMPBRGVDUBDATCAU3A, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625442339 (363h2m28.087534812s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625451045 (363h2m28.087543528s) [noise=true , address=45.91.116.5:4299 , piece_id=PTHZL456TJBBK4EKF4IGZZBOVHX2SUPZGYYJNKB3WEWMWG6FQY2Q, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625480922 (363h2m28.087573395s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 16:33:42.625487094 (363h2m28.087579567s) [noise=true , address=77.90.31.14:4662 , piece_id=AIH2FNOMJJUSO7G5NNJL5JNP7ANIL4WJCZN5C6PR2HC2WMTHBSHA, seg_pos={0 0}, upload=319361]: Long tail cancellation due to optimal threshold Oct 8 16:33:42.625501942 (363h2m28.087594425s) [seg_pos={0 0}, upload=319361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 16:33:42.625573588 (363h2m28.087666061s) [upload=319361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 16:33:42.625578127 (363h2m28.0876706s) [upload=319361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 16:33:42.936620929 (363h2m28.398713402s) [upload=319361]: Done uploading segments... Oct 8 16:33:42.936640165 (363h2m28.398732638s) [upload=319361]: Upload finished. err: <nil> Duration: 18.861784708s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)