Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [................................X..XX..XX...X..X.XXX........] duration: 5.881542609s FAIL (2025-10-07 01:09:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q) context deadline exceeded ======LOGS====== t 7 01:03:50.459807798 (323h32m35.921900281s) [upload=284743]: Starting upload Oct 7 01:03:50.459867762 (323h32m35.921960235s) [upload=284743]: Done starting upload Oct 7 01:03:50.460065419 (323h32m35.922157892s) [upload=284743]: Uploading segments... Oct 7 01:03:50.460363927 (323h32m35.92245642s) [upload=284743]: Got next segment. Inline: false Oct 7 01:03:50.460404864 (323h32m35.922497347s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:03:50.460410506 (323h32m35.922502979s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:03:50.715813017 (323h32m36.17790551s) [seg_pos={0 0}, upload=284743]: Begin upload segment... Oct 7 01:03:50.715899621 (323h32m36.177992094s) [seg_pos={0 0}, upload=284743]: Done begin upload segment. Oct 7 01:03:50.715924098 (323h32m36.178016571s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.716809483 (323h32m36.178901957s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717243621 (323h32m36.179336094s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717354882 (323h32m36.179447365s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717620609 (323h32m36.179713082s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717632671 (323h32m36.179725144s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717959705 (323h32m36.180052178s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718063302 (323h32m36.180155785s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718280355 (323h32m36.180372838s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718461950 (323h32m36.180554434s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718574995 (323h32m36.180667479s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718823789 (323h32m36.180916272s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718828328 (323h32m36.180920801s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719188033 (323h32m36.181280516s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719709937 (323h32m36.18180242s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.720907838 (323h32m36.183000321s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:52.431962504 (323h32m37.894054977s) [upload=284743]: Next returned nil segment Oct 7 01:03:52.431988694 (323h32m37.894081167s) [upload=284743]: Waiting for error group managing segments... Oct 7 01:03:52.719188280 (323h32m38.181280753s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.719818340 (323h32m38.181910823s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.720408995 (323h32m38.182501478s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.727389888 (323h32m38.189482361s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.729275599 (323h32m38.191368072s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.936259661 (323h32m38.398352144s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.987763733 (323h32m38.449856206s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.991511610 (323h32m38.453604093s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.996182033 (323h32m38.458274506s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.006584812 (323h32m38.468677295s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.059084270 (323h32m38.521176753s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.067772633 (323h32m38.529865117s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091738148 (323h32m38.553830631s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091767073 (323h32m38.553859546s) [seg_pos={0 0}, upload=284743]: Segment reached optimal threshold of 13 pieces. Oct 7 01:03:53.091898233 (323h32m38.553990716s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091909434 (323h32m38.554001917s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.091994296 (323h32m38.554086769s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091999285 (323h32m38.554091758s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092009454 (323h32m38.554101937s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection Oct 7 01:03:53.092021548 (323h32m38.554114021s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092037167 (323h32m38.55412964s) [seg_pos={0 0}, upload=284743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:03:53.092099667 (323h32m38.55419214s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:03:53.092104275 (323h32m38.554196748s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:03:53.458903252 (323h32m38.920995725s) [upload=284743]: Done uploading segments... Oct 7 01:03:53.458925414 (323h32m38.921017887s) [upload=284743]: Upload finished. err: <nil> Duration: 5.88157455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.969582551s FAIL (2025-10-07 01:09:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.969659096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.................................X..XX..XX...X..X.XXX.......] duration: 5.977477533s FAIL (2025-10-07 01:08:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q) context deadline exceeded ======LOGS====== t 7 01:03:50.459807798 (323h32m35.921900281s) [upload=284743]: Starting upload Oct 7 01:03:50.459867762 (323h32m35.921960235s) [upload=284743]: Done starting upload Oct 7 01:03:50.460065419 (323h32m35.922157892s) [upload=284743]: Uploading segments... Oct 7 01:03:50.460363927 (323h32m35.92245642s) [upload=284743]: Got next segment. Inline: false Oct 7 01:03:50.460404864 (323h32m35.922497347s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:03:50.460410506 (323h32m35.922502979s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:03:50.715813017 (323h32m36.17790551s) [seg_pos={0 0}, upload=284743]: Begin upload segment... Oct 7 01:03:50.715899621 (323h32m36.177992094s) [seg_pos={0 0}, upload=284743]: Done begin upload segment. Oct 7 01:03:50.715924098 (323h32m36.178016571s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.716809483 (323h32m36.178901957s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717243621 (323h32m36.179336094s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717354882 (323h32m36.179447365s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717620609 (323h32m36.179713082s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717632671 (323h32m36.179725144s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717959705 (323h32m36.180052178s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718063302 (323h32m36.180155785s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718280355 (323h32m36.180372838s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718461950 (323h32m36.180554434s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718574995 (323h32m36.180667479s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718823789 (323h32m36.180916272s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718828328 (323h32m36.180920801s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719188033 (323h32m36.181280516s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719709937 (323h32m36.18180242s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.720907838 (323h32m36.183000321s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:52.431962504 (323h32m37.894054977s) [upload=284743]: Next returned nil segment Oct 7 01:03:52.431988694 (323h32m37.894081167s) [upload=284743]: Waiting for error group managing segments... Oct 7 01:03:52.719188280 (323h32m38.181280753s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.719818340 (323h32m38.181910823s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.720408995 (323h32m38.182501478s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.727389888 (323h32m38.189482361s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.729275599 (323h32m38.191368072s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.936259661 (323h32m38.398352144s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.987763733 (323h32m38.449856206s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.991511610 (323h32m38.453604093s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.996182033 (323h32m38.458274506s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.006584812 (323h32m38.468677295s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.059084270 (323h32m38.521176753s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.067772633 (323h32m38.529865117s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091738148 (323h32m38.553830631s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091767073 (323h32m38.553859546s) [seg_pos={0 0}, upload=284743]: Segment reached optimal threshold of 13 pieces. Oct 7 01:03:53.091898233 (323h32m38.553990716s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091909434 (323h32m38.554001917s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.091994296 (323h32m38.554086769s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091999285 (323h32m38.554091758s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092009454 (323h32m38.554101937s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection Oct 7 01:03:53.092021548 (323h32m38.554114021s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092037167 (323h32m38.55412964s) [seg_pos={0 0}, upload=284743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:03:53.092099667 (323h32m38.55419214s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:03:53.092104275 (323h32m38.554196748s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:03:53.458903252 (323h32m38.920995725s) [upload=284743]: Done uploading segments... Oct 7 01:03:53.458925414 (323h32m38.921017887s) [upload=284743]: Upload finished. err: <nil> Duration: 5.9775012s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.804002557s FAIL (2025-10-07 01:08:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.804037593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................X.XXX...X...................] duration: 6.071229314s FAIL (2025-10-07 01:08:15 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.071262828s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................X.XXX...X..................] duration: 6.319346137s FAIL (2025-10-07 01:07:59 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.319383758s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................X..XX..XX...X..X.XXX......] duration: 5.982829726s FAIL (2025-10-07 01:07:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q) context deadline exceeded ======LOGS====== t 7 01:03:50.459807798 (323h32m35.921900281s) [upload=284743]: Starting upload Oct 7 01:03:50.459867762 (323h32m35.921960235s) [upload=284743]: Done starting upload Oct 7 01:03:50.460065419 (323h32m35.922157892s) [upload=284743]: Uploading segments... Oct 7 01:03:50.460363927 (323h32m35.92245642s) [upload=284743]: Got next segment. Inline: false Oct 7 01:03:50.460404864 (323h32m35.922497347s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:03:50.460410506 (323h32m35.922502979s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:03:50.715813017 (323h32m36.17790551s) [seg_pos={0 0}, upload=284743]: Begin upload segment... Oct 7 01:03:50.715899621 (323h32m36.177992094s) [seg_pos={0 0}, upload=284743]: Done begin upload segment. Oct 7 01:03:50.715924098 (323h32m36.178016571s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.716809483 (323h32m36.178901957s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717243621 (323h32m36.179336094s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717354882 (323h32m36.179447365s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717620609 (323h32m36.179713082s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717632671 (323h32m36.179725144s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717959705 (323h32m36.180052178s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718063302 (323h32m36.180155785s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718280355 (323h32m36.180372838s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718461950 (323h32m36.180554434s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718574995 (323h32m36.180667479s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718823789 (323h32m36.180916272s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718828328 (323h32m36.180920801s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719188033 (323h32m36.181280516s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719709937 (323h32m36.18180242s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.720907838 (323h32m36.183000321s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:52.431962504 (323h32m37.894054977s) [upload=284743]: Next returned nil segment Oct 7 01:03:52.431988694 (323h32m37.894081167s) [upload=284743]: Waiting for error group managing segments... Oct 7 01:03:52.719188280 (323h32m38.181280753s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.719818340 (323h32m38.181910823s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.720408995 (323h32m38.182501478s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.727389888 (323h32m38.189482361s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.729275599 (323h32m38.191368072s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.936259661 (323h32m38.398352144s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.987763733 (323h32m38.449856206s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.991511610 (323h32m38.453604093s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.996182033 (323h32m38.458274506s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.006584812 (323h32m38.468677295s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.059084270 (323h32m38.521176753s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.067772633 (323h32m38.529865117s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091738148 (323h32m38.553830631s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091767073 (323h32m38.553859546s) [seg_pos={0 0}, upload=284743]: Segment reached optimal threshold of 13 pieces. Oct 7 01:03:53.091898233 (323h32m38.553990716s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091909434 (323h32m38.554001917s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.091994296 (323h32m38.554086769s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091999285 (323h32m38.554091758s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092009454 (323h32m38.554101937s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection Oct 7 01:03:53.092021548 (323h32m38.554114021s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092037167 (323h32m38.55412964s) [seg_pos={0 0}, upload=284743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:03:53.092099667 (323h32m38.55419214s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:03:53.092104275 (323h32m38.554196748s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:03:53.458903252 (323h32m38.920995725s) [upload=284743]: Done uploading segments... Oct 7 01:03:53.458925414 (323h32m38.921017887s) [upload=284743]: Upload finished. err: <nil> Duration: 5.982862749s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.809356692s FAIL (2025-10-07 01:07:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.809389646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...................................X..XX..XX...X..X.XXX.....] duration: 5.985767238s FAIL (2025-10-07 01:07:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q) context deadline exceeded ======LOGS====== t 7 01:03:50.459807798 (323h32m35.921900281s) [upload=284743]: Starting upload Oct 7 01:03:50.459867762 (323h32m35.921960235s) [upload=284743]: Done starting upload Oct 7 01:03:50.460065419 (323h32m35.922157892s) [upload=284743]: Uploading segments... Oct 7 01:03:50.460363927 (323h32m35.92245642s) [upload=284743]: Got next segment. Inline: false Oct 7 01:03:50.460404864 (323h32m35.922497347s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:03:50.460410506 (323h32m35.922502979s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:03:50.715813017 (323h32m36.17790551s) [seg_pos={0 0}, upload=284743]: Begin upload segment... Oct 7 01:03:50.715899621 (323h32m36.177992094s) [seg_pos={0 0}, upload=284743]: Done begin upload segment. Oct 7 01:03:50.715924098 (323h32m36.178016571s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.716809483 (323h32m36.178901957s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717243621 (323h32m36.179336094s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717354882 (323h32m36.179447365s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717620609 (323h32m36.179713082s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717632671 (323h32m36.179725144s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717959705 (323h32m36.180052178s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718063302 (323h32m36.180155785s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718280355 (323h32m36.180372838s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718461950 (323h32m36.180554434s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718574995 (323h32m36.180667479s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718823789 (323h32m36.180916272s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718828328 (323h32m36.180920801s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719188033 (323h32m36.181280516s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719709937 (323h32m36.18180242s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.720907838 (323h32m36.183000321s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:52.431962504 (323h32m37.894054977s) [upload=284743]: Next returned nil segment Oct 7 01:03:52.431988694 (323h32m37.894081167s) [upload=284743]: Waiting for error group managing segments... Oct 7 01:03:52.719188280 (323h32m38.181280753s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.719818340 (323h32m38.181910823s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.720408995 (323h32m38.182501478s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.727389888 (323h32m38.189482361s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.729275599 (323h32m38.191368072s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.936259661 (323h32m38.398352144s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.987763733 (323h32m38.449856206s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.991511610 (323h32m38.453604093s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.996182033 (323h32m38.458274506s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.006584812 (323h32m38.468677295s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.059084270 (323h32m38.521176753s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.067772633 (323h32m38.529865117s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091738148 (323h32m38.553830631s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091767073 (323h32m38.553859546s) [seg_pos={0 0}, upload=284743]: Segment reached optimal threshold of 13 pieces. Oct 7 01:03:53.091898233 (323h32m38.553990716s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091909434 (323h32m38.554001917s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.091994296 (323h32m38.554086769s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091999285 (323h32m38.554091758s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092009454 (323h32m38.554101937s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection Oct 7 01:03:53.092021548 (323h32m38.554114021s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092037167 (323h32m38.55412964s) [seg_pos={0 0}, upload=284743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:03:53.092099667 (323h32m38.55419214s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:03:53.092104275 (323h32m38.554196748s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:03:53.458903252 (323h32m38.920995725s) [upload=284743]: Done uploading segments... Oct 7 01:03:53.458925414 (323h32m38.921017887s) [upload=284743]: Upload finished. err: <nil> Duration: 5.9858045s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................X.XXX...X.................] duration: 6.03710233s FAIL (2025-10-07 01:07:01 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.037136294s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....................................X..XX..XX...X..X.XXX....] duration: 5.979126013s FAIL (2025-10-07 01:06:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q) context deadline exceeded ======LOGS====== t 7 01:03:50.459807798 (323h32m35.921900281s) [upload=284743]: Starting upload Oct 7 01:03:50.459867762 (323h32m35.921960235s) [upload=284743]: Done starting upload Oct 7 01:03:50.460065419 (323h32m35.922157892s) [upload=284743]: Uploading segments... Oct 7 01:03:50.460363927 (323h32m35.92245642s) [upload=284743]: Got next segment. Inline: false Oct 7 01:03:50.460404864 (323h32m35.922497347s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:03:50.460410506 (323h32m35.922502979s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:03:50.715813017 (323h32m36.17790551s) [seg_pos={0 0}, upload=284743]: Begin upload segment... Oct 7 01:03:50.715899621 (323h32m36.177992094s) [seg_pos={0 0}, upload=284743]: Done begin upload segment. Oct 7 01:03:50.715924098 (323h32m36.178016571s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.716809483 (323h32m36.178901957s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717243621 (323h32m36.179336094s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717354882 (323h32m36.179447365s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717620609 (323h32m36.179713082s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717632671 (323h32m36.179725144s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717959705 (323h32m36.180052178s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718063302 (323h32m36.180155785s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718280355 (323h32m36.180372838s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718461950 (323h32m36.180554434s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718574995 (323h32m36.180667479s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718823789 (323h32m36.180916272s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718828328 (323h32m36.180920801s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719188033 (323h32m36.181280516s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719709937 (323h32m36.18180242s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.720907838 (323h32m36.183000321s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:52.431962504 (323h32m37.894054977s) [upload=284743]: Next returned nil segment Oct 7 01:03:52.431988694 (323h32m37.894081167s) [upload=284743]: Waiting for error group managing segments... Oct 7 01:03:52.719188280 (323h32m38.181280753s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.719818340 (323h32m38.181910823s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.720408995 (323h32m38.182501478s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.727389888 (323h32m38.189482361s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.729275599 (323h32m38.191368072s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.936259661 (323h32m38.398352144s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.987763733 (323h32m38.449856206s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.991511610 (323h32m38.453604093s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.996182033 (323h32m38.458274506s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.006584812 (323h32m38.468677295s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.059084270 (323h32m38.521176753s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.067772633 (323h32m38.529865117s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091738148 (323h32m38.553830631s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091767073 (323h32m38.553859546s) [seg_pos={0 0}, upload=284743]: Segment reached optimal threshold of 13 pieces. Oct 7 01:03:53.091898233 (323h32m38.553990716s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091909434 (323h32m38.554001917s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.091994296 (323h32m38.554086769s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091999285 (323h32m38.554091758s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092009454 (323h32m38.554101937s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection Oct 7 01:03:53.092021548 (323h32m38.554114021s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092037167 (323h32m38.55412964s) [seg_pos={0 0}, upload=284743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:03:53.092099667 (323h32m38.55419214s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:03:53.092104275 (323h32m38.554196748s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:03:53.458903252 (323h32m38.920995725s) [upload=284743]: Done uploading segments... Oct 7 01:03:53.458925414 (323h32m38.921017887s) [upload=284743]: Upload finished. err: <nil> Duration: 5.97915584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.722109405s FAIL (2025-10-07 01:06:42 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.722142569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................X.XXX...X................] duration: 6.253221571s FAIL (2025-10-07 01:06:23 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.253323386s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.921299781s FAIL (2025-10-07 01:06:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.921395684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....................................X..XX..XX...X..X.XXX...] duration: 7.387136018s FAIL (2025-10-07 01:06:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q) context deadline exceeded ======LOGS====== t 7 01:03:50.459807798 (323h32m35.921900281s) [upload=284743]: Starting upload Oct 7 01:03:50.459867762 (323h32m35.921960235s) [upload=284743]: Done starting upload Oct 7 01:03:50.460065419 (323h32m35.922157892s) [upload=284743]: Uploading segments... Oct 7 01:03:50.460363927 (323h32m35.92245642s) [upload=284743]: Got next segment. Inline: false Oct 7 01:03:50.460404864 (323h32m35.922497347s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:03:50.460410506 (323h32m35.922502979s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:03:50.715813017 (323h32m36.17790551s) [seg_pos={0 0}, upload=284743]: Begin upload segment... Oct 7 01:03:50.715899621 (323h32m36.177992094s) [seg_pos={0 0}, upload=284743]: Done begin upload segment. Oct 7 01:03:50.715924098 (323h32m36.178016571s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.716809483 (323h32m36.178901957s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717243621 (323h32m36.179336094s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717354882 (323h32m36.179447365s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717620609 (323h32m36.179713082s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717632671 (323h32m36.179725144s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717959705 (323h32m36.180052178s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718063302 (323h32m36.180155785s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718280355 (323h32m36.180372838s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718461950 (323h32m36.180554434s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718574995 (323h32m36.180667479s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718823789 (323h32m36.180916272s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718828328 (323h32m36.180920801s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719188033 (323h32m36.181280516s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719709937 (323h32m36.18180242s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.720907838 (323h32m36.183000321s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:52.431962504 (323h32m37.894054977s) [upload=284743]: Next returned nil segment Oct 7 01:03:52.431988694 (323h32m37.894081167s) [upload=284743]: Waiting for error group managing segments... Oct 7 01:03:52.719188280 (323h32m38.181280753s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.719818340 (323h32m38.181910823s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.720408995 (323h32m38.182501478s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.727389888 (323h32m38.189482361s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.729275599 (323h32m38.191368072s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.936259661 (323h32m38.398352144s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.987763733 (323h32m38.449856206s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.991511610 (323h32m38.453604093s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.996182033 (323h32m38.458274506s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.006584812 (323h32m38.468677295s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.059084270 (323h32m38.521176753s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.067772633 (323h32m38.529865117s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091738148 (323h32m38.553830631s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091767073 (323h32m38.553859546s) [seg_pos={0 0}, upload=284743]: Segment reached optimal threshold of 13 pieces. Oct 7 01:03:53.091898233 (323h32m38.553990716s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091909434 (323h32m38.554001917s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.091994296 (323h32m38.554086769s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091999285 (323h32m38.554091758s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092009454 (323h32m38.554101937s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection Oct 7 01:03:53.092021548 (323h32m38.554114021s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092037167 (323h32m38.55412964s) [seg_pos={0 0}, upload=284743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:03:53.092099667 (323h32m38.55419214s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:03:53.092104275 (323h32m38.554196748s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:03:53.458903252 (323h32m38.920995725s) [upload=284743]: Done uploading segments... Oct 7 01:03:53.458925414 (323h32m38.921017887s) [upload=284743]: Upload finished. err: <nil> Duration: 7.38716779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................X.XXX...X...............] duration: 7.19509919s FAIL (2025-10-07 01:05:31 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 7.195128857s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 16.061490119s FAIL (2025-10-07 01:05:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 16.06155813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......................................X..XX..XX...X..X.XXX..] duration: 6.263102361s FAIL (2025-10-07 01:04:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q) context deadline exceeded ======LOGS====== t 7 01:03:50.459807798 (323h32m35.921900281s) [upload=284743]: Starting upload Oct 7 01:03:50.459867762 (323h32m35.921960235s) [upload=284743]: Done starting upload Oct 7 01:03:50.460065419 (323h32m35.922157892s) [upload=284743]: Uploading segments... Oct 7 01:03:50.460363927 (323h32m35.92245642s) [upload=284743]: Got next segment. Inline: false Oct 7 01:03:50.460404864 (323h32m35.922497347s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:03:50.460410506 (323h32m35.922502979s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:03:50.715813017 (323h32m36.17790551s) [seg_pos={0 0}, upload=284743]: Begin upload segment... Oct 7 01:03:50.715899621 (323h32m36.177992094s) [seg_pos={0 0}, upload=284743]: Done begin upload segment. Oct 7 01:03:50.715924098 (323h32m36.178016571s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.716809483 (323h32m36.178901957s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717243621 (323h32m36.179336094s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717354882 (323h32m36.179447365s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717620609 (323h32m36.179713082s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717632671 (323h32m36.179725144s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717959705 (323h32m36.180052178s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718063302 (323h32m36.180155785s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718280355 (323h32m36.180372838s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718461950 (323h32m36.180554434s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718574995 (323h32m36.180667479s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718823789 (323h32m36.180916272s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718828328 (323h32m36.180920801s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719188033 (323h32m36.181280516s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719709937 (323h32m36.18180242s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.720907838 (323h32m36.183000321s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:52.431962504 (323h32m37.894054977s) [upload=284743]: Next returned nil segment Oct 7 01:03:52.431988694 (323h32m37.894081167s) [upload=284743]: Waiting for error group managing segments... Oct 7 01:03:52.719188280 (323h32m38.181280753s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.719818340 (323h32m38.181910823s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.720408995 (323h32m38.182501478s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.727389888 (323h32m38.189482361s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.729275599 (323h32m38.191368072s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.936259661 (323h32m38.398352144s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.987763733 (323h32m38.449856206s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.991511610 (323h32m38.453604093s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.996182033 (323h32m38.458274506s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.006584812 (323h32m38.468677295s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.059084270 (323h32m38.521176753s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.067772633 (323h32m38.529865117s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091738148 (323h32m38.553830631s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091767073 (323h32m38.553859546s) [seg_pos={0 0}, upload=284743]: Segment reached optimal threshold of 13 pieces. Oct 7 01:03:53.091898233 (323h32m38.553990716s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091909434 (323h32m38.554001917s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.091994296 (323h32m38.554086769s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091999285 (323h32m38.554091758s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092009454 (323h32m38.554101937s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection Oct 7 01:03:53.092021548 (323h32m38.554114021s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092037167 (323h32m38.55412964s) [seg_pos={0 0}, upload=284743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:03:53.092099667 (323h32m38.55419214s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:03:53.092104275 (323h32m38.554196748s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:03:53.458903252 (323h32m38.920995725s) [upload=284743]: Done uploading segments... Oct 7 01:03:53.458925414 (323h32m38.921017887s) [upload=284743]: Upload finished. err: <nil> Duration: 6.26312805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......................................X..XX..XX...X..X.XXX.] duration: 6.188095633s FAIL (2025-10-07 01:04:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q) context deadline exceeded ======LOGS====== t 7 01:03:50.459807798 (323h32m35.921900281s) [upload=284743]: Starting upload Oct 7 01:03:50.459867762 (323h32m35.921960235s) [upload=284743]: Done starting upload Oct 7 01:03:50.460065419 (323h32m35.922157892s) [upload=284743]: Uploading segments... Oct 7 01:03:50.460363927 (323h32m35.92245642s) [upload=284743]: Got next segment. Inline: false Oct 7 01:03:50.460404864 (323h32m35.922497347s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:03:50.460410506 (323h32m35.922502979s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:03:50.715813017 (323h32m36.17790551s) [seg_pos={0 0}, upload=284743]: Begin upload segment... Oct 7 01:03:50.715899621 (323h32m36.177992094s) [seg_pos={0 0}, upload=284743]: Done begin upload segment. Oct 7 01:03:50.715924098 (323h32m36.178016571s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.716809483 (323h32m36.178901957s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717243621 (323h32m36.179336094s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717354882 (323h32m36.179447365s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717620609 (323h32m36.179713082s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717632671 (323h32m36.179725144s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717959705 (323h32m36.180052178s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718063302 (323h32m36.180155785s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718280355 (323h32m36.180372838s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718461950 (323h32m36.180554434s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718574995 (323h32m36.180667479s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718823789 (323h32m36.180916272s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718828328 (323h32m36.180920801s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719188033 (323h32m36.181280516s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719709937 (323h32m36.18180242s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.720907838 (323h32m36.183000321s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:52.431962504 (323h32m37.894054977s) [upload=284743]: Next returned nil segment Oct 7 01:03:52.431988694 (323h32m37.894081167s) [upload=284743]: Waiting for error group managing segments... Oct 7 01:03:52.719188280 (323h32m38.181280753s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.719818340 (323h32m38.181910823s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.720408995 (323h32m38.182501478s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.727389888 (323h32m38.189482361s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.729275599 (323h32m38.191368072s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.936259661 (323h32m38.398352144s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.987763733 (323h32m38.449856206s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.991511610 (323h32m38.453604093s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.996182033 (323h32m38.458274506s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.006584812 (323h32m38.468677295s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.059084270 (323h32m38.521176753s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.067772633 (323h32m38.529865117s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091738148 (323h32m38.553830631s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091767073 (323h32m38.553859546s) [seg_pos={0 0}, upload=284743]: Segment reached optimal threshold of 13 pieces. Oct 7 01:03:53.091898233 (323h32m38.553990716s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091909434 (323h32m38.554001917s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.091994296 (323h32m38.554086769s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091999285 (323h32m38.554091758s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092009454 (323h32m38.554101937s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection Oct 7 01:03:53.092021548 (323h32m38.554114021s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092037167 (323h32m38.55412964s) [seg_pos={0 0}, upload=284743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:03:53.092099667 (323h32m38.55419214s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:03:53.092104275 (323h32m38.554196748s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:03:53.458903252 (323h32m38.920995725s) [upload=284743]: Done uploading segments... Oct 7 01:03:53.458925414 (323h32m38.921017887s) [upload=284743]: Upload finished. err: <nil> Duration: 6.188118977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................X.XXX...X..............] duration: 6.667197755s FAIL (2025-10-07 01:04:24 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.667230248s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 18.607963641s FAIL (2025-10-07 01:04:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 18.608000661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................................X..XX..XX...X..X.XXX] duration: 30.002955361s FAIL (2025-10-07 01:04:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q) context deadline exceeded ======LOGS====== t 7 01:03:50.459807798 (323h32m35.921900281s) [upload=284743]: Starting upload Oct 7 01:03:50.459867762 (323h32m35.921960235s) [upload=284743]: Done starting upload Oct 7 01:03:50.460065419 (323h32m35.922157892s) [upload=284743]: Uploading segments... Oct 7 01:03:50.460363927 (323h32m35.92245642s) [upload=284743]: Got next segment. Inline: false Oct 7 01:03:50.460404864 (323h32m35.922497347s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:03:50.460410506 (323h32m35.922502979s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:03:50.715813017 (323h32m36.17790551s) [seg_pos={0 0}, upload=284743]: Begin upload segment... Oct 7 01:03:50.715899621 (323h32m36.177992094s) [seg_pos={0 0}, upload=284743]: Done begin upload segment. Oct 7 01:03:50.715924098 (323h32m36.178016571s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.716809483 (323h32m36.178901957s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717243621 (323h32m36.179336094s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717354882 (323h32m36.179447365s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717620609 (323h32m36.179713082s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717632671 (323h32m36.179725144s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.717959705 (323h32m36.180052178s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718063302 (323h32m36.180155785s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718280355 (323h32m36.180372838s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718461950 (323h32m36.180554434s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718574995 (323h32m36.180667479s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718823789 (323h32m36.180916272s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.718828328 (323h32m36.180920801s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719188033 (323h32m36.181280516s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.719709937 (323h32m36.18180242s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:50.720907838 (323h32m36.183000321s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Uploading piece... Oct 7 01:03:52.431962504 (323h32m37.894054977s) [upload=284743]: Next returned nil segment Oct 7 01:03:52.431988694 (323h32m37.894081167s) [upload=284743]: Waiting for error group managing segments... Oct 7 01:03:52.719188280 (323h32m38.181280753s) [noise=true , address=145.79.227.2:4114 , piece_id=ARFXWCGWCWSRVDYBZKI63TP2DIOA32T4VQ52XVEEO7GCA5777OAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.719818340 (323h32m38.181910823s) [noise=true , address=145.79.227.16:4910 , piece_id=SRA6ZX7QUQAXDSP6WYM6SGSXSITYIHMHMLFTIPCQEXWFVRLRQWCQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.720408995 (323h32m38.182501478s) [noise=true , address=145.79.227.13:4804 , piece_id=RKEXEBQOV27CJUN2IRI3A2JMFC3AQJME6FIYMEYGWPKGKMJEOMJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.727389888 (323h32m38.189482361s) [noise=true , address=145.79.227.2:4113 , piece_id=32ST7UU6NNWBQGMFOUWVLUL44T436GY4RZRQPQCHQP35WVNWNMWQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.729275599 (323h32m38.191368072s) [noise=true , address=145.79.227.4:4187 , piece_id=O2BLXFNRNNEGN3I4P3NJZF365MBVKF2N32G35QLMYO3UIB53FHOQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.936259661 (323h32m38.398352144s) [noise=true , address=45.91.116.6:4973 , piece_id=2SDCBRJRLQPZWVDP5MXFVJ6C2WKX3HRRQ5QVUFY2FVMIJMLK3U3Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.987763733 (323h32m38.449856206s) [noise=true , address=77.90.31.15:4697 , piece_id=2U7GVBUKGXDAHV2QMQR5BTF7JDS5URTGZI2WVNACP6SQM6DWXWAA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.991511610 (323h32m38.453604093s) [noise=true , address=45.91.116.4:4938 , piece_id=DOGSDCSN2IKFISSZJ7VJQUZCASUYWUWHINLVBHO3HM5TXT4MLLHQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:52.996182033 (323h32m38.458274506s) [noise=true , address=45.91.116.12:4225 , piece_id=TERMFVWD5J6NBB6GTQWEPQAFP4WOJ6CZKV7BX633DQHSJIHEBDKA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.006584812 (323h32m38.468677295s) [noise=true , address=77.90.31.13:4626 , piece_id=7KVEAF7N57KNIUAV5OK3XGBXANPMY3VO3VUNVQ2HJOOMTZANVUJA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.059084270 (323h32m38.521176753s) [noise=true , address=77.90.31.17:4771 , piece_id=ZGDQH7ADAYHD4JVDFNCEO4DCD6JUGK6IPIJAHRNVSMESYGKYHNLA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.067772633 (323h32m38.529865117s) [noise=true , address=45.91.116.5:4299 , piece_id=N5C4XJ5HVMJRKECKCJPVTYSQXQP3KLRHMNYIAGC2M2CG6N25G5XA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091738148 (323h32m38.553830631s) [noise=true , address=45.91.116.14:4190 , piece_id=3IIOMD2FG6WQQ4SGF6RAAISUW4I4XPOLCZUTLPOXB6BNMYKIS26Q, seg_pos={0 0}, upload=284743]: Done uploading piece. err: <nil> Oct 7 01:03:53.091767073 (323h32m38.553859546s) [seg_pos={0 0}, upload=284743]: Segment reached optimal threshold of 13 pieces. Oct 7 01:03:53.091898233 (323h32m38.553990716s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091909434 (323h32m38.554001917s) [noise=true , address=45.91.116.6:4974 , piece_id=M4JHTDMJB5SM2XZPBOS3NPTJ3RJFNPBP344LZA3FT4S4JF3AAA7A, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.091994296 (323h32m38.554086769s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:53.091999285 (323h32m38.554091758s) [noise=true , address=45.91.116.9:4263 , piece_id=4F5ZH6FHPBPTDFKH5YMYVDCOXDDQSYQGN4E65XNZ6GZIWTT6H5GQ, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092009454 (323h32m38.554101937s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48488->45.91.116.7:4009: use of closed network connection Oct 7 01:03:53.092021548 (323h32m38.554114021s) [noise=true , address=45.91.116.7:4009 , piece_id=3SIGONZVXA3AQKHQAQLJORCLN3REBC6EBJOEAPYMFJ3G7E22PEDA, seg_pos={0 0}, upload=284743]: Long tail cancellation due to optimal threshold Oct 7 01:03:53.092037167 (323h32m38.55412964s) [seg_pos={0 0}, upload=284743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:03:53.092099667 (323h32m38.55419214s) [upload=284743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:03:53.092104275 (323h32m38.554196748s) [upload=284743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:03:53.458903252 (323h32m38.920995725s) [upload=284743]: Done uploading segments... Oct 7 01:03:53.458925414 (323h32m38.921017887s) [upload=284743]: Upload finished. err: <nil> Duration: 30.002989737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.298439586s FAIL (2025-10-07 01:03:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 6.29847849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........................................X..XX..XX...X..X.XX] duration: 30.002569295s FAIL (2025-10-07 01:03:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 01:03:10.668688330 (323h31m56.130780803s) [upload=284732]: Starting upload Oct 7 01:03:10.668764384 (323h31m56.130856857s) [upload=284732]: Done starting upload Oct 7 01:03:10.668880837 (323h31m56.13097332s) [upload=284732]: Uploading segments... Oct 7 01:03:10.668952593 (323h31m56.131045076s) [upload=284732]: Got next segment. Inline: false Oct 7 01:03:10.668983321 (323h31m56.131075804s) [upload=284732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:03:10.668988371 (323h31m56.131080854s) [upload=284732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:03:10.972981755 (323h31m56.435074238s) [seg_pos={0 0}, upload=284732]: Begin upload segment... Oct 7 01:03:10.973063850 (323h31m56.435156334s) [seg_pos={0 0}, upload=284732]: Done begin upload segment. Oct 7 01:03:10.973094960 (323h31m56.435187433s) [noise=true , address=77.90.31.13:4628 , piece_id=XB56WWKP3DIVWWJJ3BNE7Q2ETBVZDFRW2ELGVJ2ZOYMHFCMH4LHQ, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.973101523 (323h31m56.435193996s) [noise=true , address=45.91.116.13:4153 , piece_id=HQQ6NJ7ERFXC6YLXDABMFANIAXAV6JYYBD3SY7SPWZXXTLPH5SPQ, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.973253392 (323h31m56.435345875s) [noise=true , address=45.91.116.6:4974 , piece_id=ECS5RHB3ESRGGGXO3BUZT7CWPL7CEHM4F32DDU5OL3KIY5AYA62A, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.973842965 (323h31m56.435935448s) [noise=true , address=45.91.116.5:4299 , piece_id=WSQFWQKDGTMVE7MQIBDG75PJZ3J24SJE5P6ICEMPN5MPY42RCLJQ, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.974295617 (323h31m56.4363881s) [noise=true , address=77.90.31.12:4592 , piece_id=H6O5AD3CJDPOSZ24FKK3PSXOHHCUDTZTCCDD7XSZYLIJIZ2DW3GA, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.974689486 (323h31m56.436781969s) [noise=true , address=145.79.227.2:4113 , piece_id=36ZWAMQKDVJEKS4TGSCG2NW45LXUY22AYZ4AHAFAW5KAZSKCO4XA, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.975102162 (323h31m56.437194645s) [noise=true , address=145.79.227.14:4838 , piece_id=WVAQUBOW3YKQU3VOJWD4STLJES6FYMCQNUWZAJREBNQHWKPTHE5Q, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.975540046 (323h31m56.43763253s) [noise=true , address=145.79.227.16:4910 , piece_id=CY26W4E2NZOBCA327IW6F42XPG2OYQ5XRSKO2OPZDOKTLEHBJUGA, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.976139087 (323h31m56.43823157s) [noise=true , address=145.79.227.15:4875 , piece_id=4RSRU2J6DRKXFPYOCAWJ3JZ5LW3GWE3OUZ63V7OBNQKEDIENI2TQ, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.976666150 (323h31m56.438758623s) [noise=true , address=77.90.31.13:4626 , piece_id=HSZZHCYNTDMKF7YFUJZ2LTKCFQT6MTWOS6AFEX7QNGOXAFDS5IAA, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.977171733 (323h31m56.439264216s) [noise=true , address=45.91.116.11:4118 , piece_id=DCTROR36FU7YLJ2ZQLSM4KWPQLFA2OZ53KSOGG55I3EYBOKP6UEA, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.978701926 (323h31m56.440794409s) [noise=true , address=77.90.31.18:4805 , piece_id=5FR2DW3MRKXCO6WZY7JDG6WHWZSFB5JYT35EHVTVFV7B2CKSNPUQ, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.979732910 (323h31m56.441825393s) [noise=true , address=145.79.227.3:4151 , piece_id=5E3EH5BSCZYLT5J3C4C4GGCJBAUPRV3AWBK4PMCL2ME3RCZ2DW7Q, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.984515557 (323h31m56.44660805s) [noise=true , address=145.79.227.18:4006 , piece_id=FDWYUNA2VTB5EJ3QREUYLN2ZZPII44KNSPAS67KVEKSILQ5KM64Q, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.988580417 (323h31m56.4506729s) [noise=true , address=145.79.227.13:4801 , piece_id=XPXSSAZUAUVDGFZUBM2URP2WKFKO6TD7WMZ7JAEVFFSXVPNZ4TRA, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:10.989577776 (323h31m56.45167026s) [noise=true , address=145.79.227.7:4294 , piece_id=TDZLFNXW2BQXKCWRBDOFS4MKUENH6UASXN5SIQCKYSRUR54W5XDA, seg_pos={0 0}, upload=284732]: Uploading piece... Oct 7 01:03:12.690364933 (323h31m58.152457406s) [upload=284732]: Next returned nil segment Oct 7 01:03:12.690384219 (323h31m58.152476692s) [upload=284732]: Waiting for error group managing segments... Oct 7 01:03:12.980886234 (323h31m58.442978717s) [noise=true , address=145.79.227.16:4910 , piece_id=CY26W4E2NZOBCA327IW6F42XPG2OYQ5XRSKO2OPZDOKTLEHBJUGA, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:12.980985092 (323h31m58.443077585s) [noise=true , address=145.79.227.3:4151 , piece_id=5E3EH5BSCZYLT5J3C4C4GGCJBAUPRV3AWBK4PMCL2ME3RCZ2DW7Q, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:12.988283450 (323h31m58.450375923s) [noise=true , address=145.79.227.18:4006 , piece_id=FDWYUNA2VTB5EJ3QREUYLN2ZZPII44KNSPAS67KVEKSILQ5KM64Q, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:12.992386762 (323h31m58.454479245s) [noise=true , address=145.79.227.15:4875 , piece_id=4RSRU2J6DRKXFPYOCAWJ3JZ5LW3GWE3OUZ63V7OBNQKEDIENI2TQ, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:12.994568938 (323h31m58.456661411s) [noise=true , address=145.79.227.14:4838 , piece_id=WVAQUBOW3YKQU3VOJWD4STLJES6FYMCQNUWZAJREBNQHWKPTHE5Q, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:12.995937554 (323h31m58.458030037s) [noise=true , address=145.79.227.2:4113 , piece_id=36ZWAMQKDVJEKS4TGSCG2NW45LXUY22AYZ4AHAFAW5KAZSKCO4XA, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:13.001544841 (323h31m58.463637344s) [noise=true , address=145.79.227.13:4801 , piece_id=XPXSSAZUAUVDGFZUBM2URP2WKFKO6TD7WMZ7JAEVFFSXVPNZ4TRA, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:13.010550159 (323h31m58.472642642s) [noise=true , address=145.79.227.7:4294 , piece_id=TDZLFNXW2BQXKCWRBDOFS4MKUENH6UASXN5SIQCKYSRUR54W5XDA, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:13.080770560 (323h31m58.542863033s) [noise=true , address=77.90.31.12:4592 , piece_id=H6O5AD3CJDPOSZ24FKK3PSXOHHCUDTZTCCDD7XSZYLIJIZ2DW3GA, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:13.093437571 (323h31m58.555530044s) [noise=true , address=45.91.116.13:4153 , piece_id=HQQ6NJ7ERFXC6YLXDABMFANIAXAV6JYYBD3SY7SPWZXXTLPH5SPQ, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:13.167109374 (323h31m58.629201857s) [noise=true , address=45.91.116.6:4974 , piece_id=ECS5RHB3ESRGGGXO3BUZT7CWPL7CEHM4F32DDU5OL3KIY5AYA62A, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:13.238595295 (323h31m58.700687768s) [noise=true , address=77.90.31.13:4626 , piece_id=HSZZHCYNTDMKF7YFUJZ2LTKCFQT6MTWOS6AFEX7QNGOXAFDS5IAA, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:13.241085087 (323h31m58.70317757s) [noise=true , address=45.91.116.5:4299 , piece_id=WSQFWQKDGTMVE7MQIBDG75PJZ3J24SJE5P6ICEMPN5MPY42RCLJQ, seg_pos={0 0}, upload=284732]: Done uploading piece. err: <nil> Oct 7 01:03:13.241106667 (323h31m58.70319915s) [seg_pos={0 0}, upload=284732]: Segment reached optimal threshold of 13 pieces. Oct 7 01:03:13.241272413 (323h31m58.703364896s) [noise=true , address=77.90.31.13:4628 , piece_id=XB56WWKP3DIVWWJJ3BNE7Q2ETBVZDFRW2ELGVJ2ZOYMHFCMH4LHQ, seg_pos={0 0}, upload=284732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:13.241281951 (323h31m58.703374434s) [noise=true , address=77.90.31.13:4628 , piece_id=XB56WWKP3DIVWWJJ3BNE7Q2ETBVZDFRW2ELGVJ2ZOYMHFCMH4LHQ, seg_pos={0 0}, upload=284732]: Long tail cancellation due to optimal threshold Oct 7 01:03:13.241381260 (323h31m58.703473743s) [noise=true , address=77.90.31.18:4805 , piece_id=5FR2DW3MRKXCO6WZY7JDG6WHWZSFB5JYT35EHVTVFV7B2CKSNPUQ, seg_pos={0 0}, upload=284732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Oct 7 01:03:13.241388974 (323h31m58.703481458s) [noise=true , address=77.90.31.18:4805 , piece_id=5FR2DW3MRKXCO6WZY7JDG6WHWZSFB5JYT35EHVTVFV7B2CKSNPUQ, seg_pos={0 0}, upload=284732]: Long tail cancellation due to optimal threshold Oct 7 01:03:13.241630434 (323h31m58.703722917s) [noise=true , address=45.91.116.11:4118 , piece_id=DCTROR36FU7YLJ2ZQLSM4KWPQLFA2OZ53KSOGG55I3EYBOKP6UEA, seg_pos={0 0}, upload=284732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: read tcp 188.166.225.170:38516->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 01:03:13.241640584 (323h31m58.703733067s) [noise=true , address=45.91.116.11:4118 , piece_id=DCTROR36FU7YLJ2ZQLSM4KWPQLFA2OZ53KSOGG55I3EYBOKP6UEA, seg_pos={0 0}, upload=284732]: Long tail cancellation due to optimal threshold Oct 7 01:03:13.241660662 (323h31m58.703753135s) [seg_pos={0 0}, upload=284732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:03:13.241742207 (323h31m58.70383469s) [upload=284732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:03:13.241748209 (323h31m58.703840692s) [upload=284732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:03:13.670500817 (323h31m59.1325933s) [upload=284732]: Done uploading segments... Oct 7 01:03:13.670528959 (323h31m59.132621442s) [upload=284732]: Upload finished. err: <nil> Duration: 30.002763054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................X.XXX...X.............] duration: 6.488168722s FAIL (2025-10-07 01:03:36 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.488202387s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................................X..XX..XX...X..X.X] duration: 30.003233153s FAIL (2025-10-07 01:03:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (323h31m24.144116101s) [upload=284723]: Done starting upload Oct 7 01:02:38.682152714 (323h31m24.144245197s) [upload=284723]: Uploading segments... Oct 7 01:02:38.682273915 (323h31m24.144366398s) [upload=284723]: Got next segment. Inline: false Oct 7 01:02:38.682317649 (323h31m24.144410132s) [upload=284723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:02:38.682324081 (323h31m24.144416564s) [upload=284723]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:02:38.965008370 (323h31m24.427100853s) [seg_pos={0 0}, upload=284723]: Begin upload segment... Oct 7 01:02:38.965060870 (323h31m24.427153343s) [seg_pos={0 0}, upload=284723]: Done begin upload segment. Oct 7 01:02:38.965087241 (323h31m24.427179714s) [noise=true , address=45.91.116.13:4155 , piece_id=HUWMFUHTVME5YO5GTTBFC6HTRTXUHWT6FKXVRU2YVCOKVZISPXXA, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.965133629 (323h31m24.427226112s) [noise=true , address=145.79.227.11:4729 , piece_id=TVKENJPDD6E7MLTRXOVIDZL2EBNIIZAQDFWDWFBSHDMLJRFCDULQ, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.965527058 (323h31m24.427619531s) [noise=true , address=77.90.31.15:4699 , piece_id=LGZ5MHCSU2RIWGW7CGYN7QS5BXCSDTYFKZX27XXAGVQYWDEPHIYA, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.965858649 (323h31m24.427951133s) [noise=true , address=77.90.31.11:4555 , piece_id=DKCT34J7HNPR6MVOXPBMYW73CXU3TRUC6DATV2RXQKFCERRUTIVQ, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.966178488 (323h31m24.428270971s) [noise=true , address=77.90.31.12:4590 , piece_id=QU3Q5U4F3NOCF5VVT54C4LESUVPEI72TMD5XDZ6HV4RSU7DNVVHA, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.966478871 (323h31m24.428571354s) [noise=true , address=45.91.116.6:4975 , piece_id=KX6ULT5SRDFNKFKGDRSTJ55GA6BXWR5TJVP57PVZW3KIUO2IGI3Q, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.966788449 (323h31m24.428880922s) [noise=true , address=145.79.227.5:4223 , piece_id=TEG65D36QYGA4N4OWAYS2E7BHO3QZKL7NMW4B5RGJCCA5FBEG25Q, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.967091276 (323h31m24.429183749s) [noise=true , address=145.79.227.18:4006 , piece_id=QDDBKFSZGP44DHGPOF62U3SXNKB7OMM4HFL3ODGJFVRCLQAV4QBQ, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.967462283 (323h31m24.429554766s) [noise=true , address=45.91.116.4:4937 , piece_id=UPMUPO247PKVSSRDE2ZJVOL4JDUVYMWVBM7TTZUURC2I2RQCLPYQ, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.968955957 (323h31m24.43104844s) [noise=true , address=77.90.31.17:4771 , piece_id=WJNWFXQVKGDDZLCEBBHFDKFSZG6QELB7TDHGOBVXOGDZ6PXCEU2Q, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.970869731 (323h31m24.432962215s) [noise=true , address=145.79.227.14:4839 , piece_id=JHF554BEEWPXBYHMDQMFRIGSSGHZBNMEELJ4SY34NKRFY6C3QJMQ, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.972589215 (323h31m24.434681698s) [noise=true , address=145.79.227.7:4295 , piece_id=TVMWXQ34EONADCBH6JGYRMDM6ZN2SJ2RUA5CZNOUL3UX4D4BZTGQ, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.973966669 (323h31m24.436059152s) [noise=true , address=45.91.116.12:4227 , piece_id=5G3BA6TYHBACWZGVCUJQPDBPVZ2ASBOVO63BKF6NX6HN3ELF556A, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.975286161 (323h31m24.437378644s) [noise=true , address=45.91.116.5:4299 , piece_id=NBQE24V75ZTGOMX2ZAT5OWH32BTJXWDXWLCTJRACMS6L7QW7LNWA, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.976437433 (323h31m24.438529916s) [noise=true , address=45.91.116.13:4154 , piece_id=UMDOND5ISC6SM3DA5JH6NKORPMNB7QEK26LZ5N77PDUPULPPJSGQ, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:38.977080157 (323h31m24.43917263s) [noise=true , address=45.91.116.8:4047 , piece_id=IFNBRHWCT3BLWTROVPZM6GWYLT5GSLEBOBJTVTH52MRVEAFMLKSA, seg_pos={0 0}, upload=284723]: Uploading piece... Oct 7 01:02:40.677522345 (323h31m26.139614828s) [upload=284723]: Next returned nil segment Oct 7 01:02:40.677545299 (323h31m26.139637782s) [upload=284723]: Waiting for error group managing segments... Oct 7 01:02:40.961312389 (323h31m26.423404872s) [noise=true , address=145.79.227.5:4223 , piece_id=TEG65D36QYGA4N4OWAYS2E7BHO3QZKL7NMW4B5RGJCCA5FBEG25Q, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:40.968100084 (323h31m26.430192567s) [noise=true , address=145.79.227.7:4295 , piece_id=TVMWXQ34EONADCBH6JGYRMDM6ZN2SJ2RUA5CZNOUL3UX4D4BZTGQ, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:40.975071599 (323h31m26.437164082s) [noise=true , address=145.79.227.18:4006 , piece_id=QDDBKFSZGP44DHGPOF62U3SXNKB7OMM4HFL3ODGJFVRCLQAV4QBQ, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:40.993911434 (323h31m26.456003907s) [noise=true , address=145.79.227.14:4839 , piece_id=JHF554BEEWPXBYHMDQMFRIGSSGHZBNMEELJ4SY34NKRFY6C3QJMQ, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:40.996635280 (323h31m26.458727763s) [noise=true , address=145.79.227.11:4729 , piece_id=TVKENJPDD6E7MLTRXOVIDZL2EBNIIZAQDFWDWFBSHDMLJRFCDULQ, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:41.118628496 (323h31m26.580720969s) [noise=true , address=45.91.116.6:4975 , piece_id=KX6ULT5SRDFNKFKGDRSTJ55GA6BXWR5TJVP57PVZW3KIUO2IGI3Q, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:41.148571973 (323h31m26.610664456s) [noise=true , address=77.90.31.15:4699 , piece_id=LGZ5MHCSU2RIWGW7CGYN7QS5BXCSDTYFKZX27XXAGVQYWDEPHIYA, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:41.182289056 (323h31m26.644381539s) [noise=true , address=45.91.116.4:4937 , piece_id=UPMUPO247PKVSSRDE2ZJVOL4JDUVYMWVBM7TTZUURC2I2RQCLPYQ, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:41.195932826 (323h31m26.658025309s) [noise=true , address=77.90.31.11:4555 , piece_id=DKCT34J7HNPR6MVOXPBMYW73CXU3TRUC6DATV2RXQKFCERRUTIVQ, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:41.228309146 (323h31m26.690401629s) [noise=true , address=45.91.116.5:4299 , piece_id=NBQE24V75ZTGOMX2ZAT5OWH32BTJXWDXWLCTJRACMS6L7QW7LNWA, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:41.249618242 (323h31m26.711710725s) [noise=true , address=45.91.116.13:4154 , piece_id=UMDOND5ISC6SM3DA5JH6NKORPMNB7QEK26LZ5N77PDUPULPPJSGQ, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:41.261793666 (323h31m26.723886159s) [noise=true , address=45.91.116.12:4227 , piece_id=5G3BA6TYHBACWZGVCUJQPDBPVZ2ASBOVO63BKF6NX6HN3ELF556A, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:41.346335328 (323h31m26.808427801s) [noise=true , address=77.90.31.17:4771 , piece_id=WJNWFXQVKGDDZLCEBBHFDKFSZG6QELB7TDHGOBVXOGDZ6PXCEU2Q, seg_pos={0 0}, upload=284723]: Done uploading piece. err: <nil> Oct 7 01:02:41.346355606 (323h31m26.808448089s) [seg_pos={0 0}, upload=284723]: Segment reached optimal threshold of 13 pieces. Oct 7 01:02:41.346465015 (323h31m26.808557488s) [noise=true , address=45.91.116.13:4155 , piece_id=HUWMFUHTVME5YO5GTTBFC6HTRTXUHWT6FKXVRU2YVCOKVZISPXXA, seg_pos={0 0}, upload=284723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Oct 7 01:02:41.346472098 (323h31m26.808564581s) [noise=true , address=45.91.116.13:4155 , piece_id=HUWMFUHTVME5YO5GTTBFC6HTRTXUHWT6FKXVRU2YVCOKVZISPXXA, seg_pos={0 0}, upload=284723]: Long tail cancellation due to optimal threshold Oct 7 01:02:41.346587638 (323h31m26.808680121s) [noise=true , address=45.91.116.8:4047 , piece_id=IFNBRHWCT3BLWTROVPZM6GWYLT5GSLEBOBJTVTH52MRVEAFMLKSA, seg_pos={0 0}, upload=284723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:57042->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:57042->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57042->45.91.116.8:4047: use of closed network connection Oct 7 01:02:41.346613327 (323h31m26.80870583s) [noise=true , address=45.91.116.8:4047 , piece_id=IFNBRHWCT3BLWTROVPZM6GWYLT5GSLEBOBJTVTH52MRVEAFMLKSA, seg_pos={0 0}, upload=284723]: Long tail cancellation due to optimal threshold Oct 7 01:02:41.346715301 (323h31m26.808807774s) [noise=true , address=77.90.31.12:4590 , piece_id=QU3Q5U4F3NOCF5VVT54C4LESUVPEI72TMD5XDZ6HV4RSU7DNVVHA, seg_pos={0 0}, upload=284723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: read tcp 188.166.225.170:38890->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 01:02:41.346733545 (323h31m26.808826019s) [noise=true , address=77.90.31.12:4590 , piece_id=QU3Q5U4F3NOCF5VVT54C4LESUVPEI72TMD5XDZ6HV4RSU7DNVVHA, seg_pos={0 0}, upload=284723]: Long tail cancellation due to optimal threshold Oct 7 01:02:41.346752151 (323h31m26.808844624s) [seg_pos={0 0}, upload=284723]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:02:41.346829899 (323h31m26.808922382s) [upload=284723]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:02:41.346834929 (323h31m26.808927412s) [upload=284723]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:02:42.534381017 (323h31m27.99647349s) [upload=284723]: Done uploading segments... Oct 7 01:02:42.534401736 (323h31m27.996494219s) [upload=284723]: Upload finished. err: <nil> Duration: 30.003263429s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 15.814307239s FAIL (2025-10-07 01:03:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 15.814349931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................X.XXX...X............] duration: 7.115449925s FAIL (2025-10-07 01:02:45 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 7.115480602s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 8.985278241s FAIL (2025-10-07 01:02:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 8.98530913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.876271096s FAIL (2025-10-07 01:01:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.876299911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................................X..XX..XX...X..X.] duration: 6.190111261s FAIL (2025-10-07 01:01:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: TQ5EOVF2WRBCJACPJM3737IOCFAR6NAYWDYIVUFZY6U2YU3JO2SA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: HMMJUK3GQH73MREUCXLUQJBHTZAOADUJCVWFCR3AJBTCP4GRO5AQ) context deadline exceeded ======LOGS====== ct 7 01:01:10.121839009 (323h29m55.583931492s) [upload=284702]: Starting upload Oct 7 01:01:10.121929321 (323h29m55.584021804s) [upload=284702]: Done starting upload Oct 7 01:01:10.122196790 (323h29m55.584289273s) [upload=284702]: Uploading segments... Oct 7 01:01:10.122261994 (323h29m55.584354478s) [upload=284702]: Got next segment. Inline: false Oct 7 01:01:10.122302772 (323h29m55.584395255s) [upload=284702]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:01:10.122308583 (323h29m55.584401076s) [upload=284702]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:01:10.411338415 (323h29m55.873430898s) [seg_pos={0 0}, upload=284702]: Begin upload segment... Oct 7 01:01:10.411397707 (323h29m55.87349019s) [seg_pos={0 0}, upload=284702]: Done begin upload segment. Oct 7 01:01:10.411423877 (323h29m55.87351635s) [noise=true , address=145.79.227.4:4186 , piece_id=N7TEHBR7IWNLBUILH332AC7WNSJNZ4XYDS7AX5GCECPCPI7FB4HA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.411471208 (323h29m55.873563681s) [noise=true , address=145.79.227.14:4839 , piece_id=LNOEB3X2XDTT47M6GATWJWVYA3R62L5XMXLZKEJ374GKPUM4KTRQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.412301959 (323h29m55.874394442s) [noise=true , address=145.79.227.19:4077 , piece_id=CLZNAOMAJXXZLVODTRFFOL52LUO5BQQJL224AMBPHGWT4AMSME3A, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413077777 (323h29m55.87517025s) [noise=true , address=45.91.116.5:4299 , piece_id=TQ5EOVF2WRBCJACPJM3737IOCFAR6NAYWDYIVUFZY6U2YU3JO2SA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413115959 (323h29m55.875208432s) [noise=true , address=77.90.31.14:4663 , piece_id=HMMJUK3GQH73MREUCXLUQJBHTZAOADUJCVWFCR3AJBTCP4GRO5AQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413422523 (323h29m55.875515006s) [noise=true , address=77.90.31.16:4733 , piece_id=CKP2NN7B7J2PAGORMCQ42MGISFEQQUQLIQR32CVD5YBVLVC2L4YA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413833645 (323h29m55.875926128s) [noise=true , address=77.90.31.18:4806 , piece_id=E7VGWTIYKDDKBRKWUGAINRUEJ47EBJYKEGS5X5HNOCKHZT5ZR45A, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413966208 (323h29m55.876058691s) [noise=true , address=145.79.227.17:4972 , piece_id=GTLUNX4O7C5EEIUP25RZJWGQ2OBTHOBBZDWM76QAQMCRLTRR2N2Q, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.414258555 (323h29m55.876351028s) [noise=true , address=145.79.227.7:4293 , piece_id=OQQFJVQBPFQPMKCJOJLPK3VWT5ZQRZCFNOGPWOGX4ZD7BXKM6GHA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.414710566 (323h29m55.876803049s) [noise=true , address=77.90.31.11:4553 , piece_id=3NDPW2SUKFOGTUHP7C2OOICBT6DND77SHIN6GX3SBUOJCQKKHNMA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.414886451 (323h29m55.876978924s) [noise=true , address=145.79.227.15:4873 , piece_id=RALEQ5GZAQ4UFZUKOCQZLLWOJBXWXP5A2RDQHNCO226NK5WFAYQA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.415062666 (323h29m55.877155149s) [noise=true , address=145.79.227.3:4150 , piece_id=KCZXJ26XHDVDXR7R5NVZKJEASLQDH4ASUZE2CYHSJWW7MN3J5LHQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.415419926 (323h29m55.877512409s) [noise=true , address=145.79.227.13:4803 , piece_id=7KCTQU6ZBG6FZQHSQN64PXJUXDNCHUCDQRFUNCC5NNJL3MF4FQSQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.415697455 (323h29m55.877789948s) [noise=true , address=145.79.227.19:4078 , piece_id=5CBQOJN4YUTUONPOXK2TNPNYLBEYPBJJW3UFZ4Z55K2TFJ6L2NCA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.416700565 (323h29m55.878793038s) [noise=true , address=77.90.31.19:4842 , piece_id=NNKDBWZPJJEB7SLFEBUP5LO4HBHQ7MSPPUHV6CQIDRIHJV6OGS6A, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.417369609 (323h29m55.879462092s) [noise=true , address=45.91.116.12:4226 , piece_id=V63GMYNF6RXLUHYPLSYX4CGQF4BBPF7IOFVXSFT4N77T7RCGY4LA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:12.119016842 (323h29m57.581109325s) [upload=284702]: Next returned nil segment Oct 7 01:01:12.119114698 (323h29m57.581207201s) [upload=284702]: Waiting for error group managing segments... Oct 7 01:01:12.398844259 (323h29m57.860936742s) [noise=true , address=145.79.227.19:4077 , piece_id=CLZNAOMAJXXZLVODTRFFOL52LUO5BQQJL224AMBPHGWT4AMSME3A, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.402151106 (323h29m57.864243579s) [noise=true , address=145.79.227.14:4839 , piece_id=LNOEB3X2XDTT47M6GATWJWVYA3R62L5XMXLZKEJ374GKPUM4KTRQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.405187528 (323h29m57.867280011s) [noise=true , address=145.79.227.3:4150 , piece_id=KCZXJ26XHDVDXR7R5NVZKJEASLQDH4ASUZE2CYHSJWW7MN3J5LHQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.421155123 (323h29m57.883247596s) [noise=true , address=145.79.227.15:4873 , piece_id=RALEQ5GZAQ4UFZUKOCQZLLWOJBXWXP5A2RDQHNCO226NK5WFAYQA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.421662399 (323h29m57.883754883s) [noise=true , address=145.79.227.13:4803 , piece_id=7KCTQU6ZBG6FZQHSQN64PXJUXDNCHUCDQRFUNCC5NNJL3MF4FQSQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.423239743 (323h29m57.885332226s) [noise=true , address=145.79.227.17:4972 , piece_id=GTLUNX4O7C5EEIUP25RZJWGQ2OBTHOBBZDWM76QAQMCRLTRR2N2Q, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.425523362 (323h29m57.887615835s) [noise=true , address=145.79.227.19:4078 , piece_id=5CBQOJN4YUTUONPOXK2TNPNYLBEYPBJJW3UFZ4Z55K2TFJ6L2NCA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.425952498 (323h29m57.888044971s) [noise=true , address=145.79.227.4:4186 , piece_id=N7TEHBR7IWNLBUILH332AC7WNSJNZ4XYDS7AX5GCECPCPI7FB4HA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.431251198 (323h29m57.893343681s) [noise=true , address=145.79.227.7:4293 , piece_id=OQQFJVQBPFQPMKCJOJLPK3VWT5ZQRZCFNOGPWOGX4ZD7BXKM6GHA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.636922390 (323h29m58.099014873s) [noise=true , address=77.90.31.11:4553 , piece_id=3NDPW2SUKFOGTUHP7C2OOICBT6DND77SHIN6GX3SBUOJCQKKHNMA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.683157060 (323h29m58.145249543s) [noise=true , address=77.90.31.19:4842 , piece_id=NNKDBWZPJJEB7SLFEBUP5LO4HBHQ7MSPPUHV6CQIDRIHJV6OGS6A, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.683342783 (323h29m58.145435256s) [noise=true , address=45.91.116.5:4299 , piece_id=TQ5EOVF2WRBCJACPJM3737IOCFAR6NAYWDYIVUFZY6U2YU3JO2SA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.709183959 (323h29m58.171276442s) [noise=true , address=77.90.31.14:4663 , piece_id=HMMJUK3GQH73MREUCXLUQJBHTZAOADUJCVWFCR3AJBTCP4GRO5AQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.709211742 (323h29m58.171304235s) [seg_pos={0 0}, upload=284702]: Segment reached optimal threshold of 13 pieces. Oct 7 01:01:12.709372308 (323h29m58.171464791s) [noise=true , address=77.90.31.18:4806 , piece_id=E7VGWTIYKDDKBRKWUGAINRUEJ47EBJYKEGS5X5HNOCKHZT5ZR45A, seg_pos={0 0}, upload=284702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:38918->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:38918->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38918->77.90.31.18:4806: use of closed network connection Oct 7 01:01:12.709384090 (323h29m58.171476573s) [noise=true , address=77.90.31.18:4806 , piece_id=E7VGWTIYKDDKBRKWUGAINRUEJ47EBJYKEGS5X5HNOCKHZT5ZR45A, seg_pos={0 0}, upload=284702]: Long tail cancellation due to optimal threshold Oct 7 01:01:12.709527122 (323h29m58.171619595s) [noise=true , address=45.91.116.12:4226 , piece_id=V63GMYNF6RXLUHYPLSYX4CGQF4BBPF7IOFVXSFT4N77T7RCGY4LA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 7 01:01:12.709535739 (323h29m58.171628222s) [noise=true , address=45.91.116.12:4226 , piece_id=V63GMYNF6RXLUHYPLSYX4CGQF4BBPF7IOFVXSFT4N77T7RCGY4LA, seg_pos={0 0}, upload=284702]: Long tail cancellation due to optimal threshold Oct 7 01:01:12.709614038 (323h29m58.171706521s) [noise=true , address=77.90.31.16:4733 , piece_id=CKP2NN7B7J2PAGORMCQ42MGISFEQQUQLIQR32CVD5YBVLVC2L4YA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 7 01:01:12.709624408 (323h29m58.171716891s) [noise=true , address=77.90.31.16:4733 , piece_id=CKP2NN7B7J2PAGORMCQ42MGISFEQQUQLIQR32CVD5YBVLVC2L4YA, seg_pos={0 0}, upload=284702]: Long tail cancellation due to optimal threshold Oct 7 01:01:12.709639517 (323h29m58.171732s) [seg_pos={0 0}, upload=284702]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:01:12.709732493 (323h29m58.171824976s) [upload=284702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:01:12.709738976 (323h29m58.171831459s) [upload=284702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:01:13.360148075 (323h29m58.822240558s) [upload=284702]: Done uploading segments... Oct 7 01:01:13.360170197 (323h29m58.82226267s) [upload=284702]: Upload finished. err: <nil> Duration: 6.190142761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................X.XXX...X...........] duration: 6.385753207s FAIL (2025-10-07 01:01:40 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.385776092s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................................X..XX..XX...X..X] duration: 30.002397277s FAIL (2025-10-07 01:01:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: TQ5EOVF2WRBCJACPJM3737IOCFAR6NAYWDYIVUFZY6U2YU3JO2SA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: HMMJUK3GQH73MREUCXLUQJBHTZAOADUJCVWFCR3AJBTCP4GRO5AQ) context deadline exceeded ======LOGS====== ct 7 01:01:10.121839009 (323h29m55.583931492s) [upload=284702]: Starting upload Oct 7 01:01:10.121929321 (323h29m55.584021804s) [upload=284702]: Done starting upload Oct 7 01:01:10.122196790 (323h29m55.584289273s) [upload=284702]: Uploading segments... Oct 7 01:01:10.122261994 (323h29m55.584354478s) [upload=284702]: Got next segment. Inline: false Oct 7 01:01:10.122302772 (323h29m55.584395255s) [upload=284702]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:01:10.122308583 (323h29m55.584401076s) [upload=284702]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:01:10.411338415 (323h29m55.873430898s) [seg_pos={0 0}, upload=284702]: Begin upload segment... Oct 7 01:01:10.411397707 (323h29m55.87349019s) [seg_pos={0 0}, upload=284702]: Done begin upload segment. Oct 7 01:01:10.411423877 (323h29m55.87351635s) [noise=true , address=145.79.227.4:4186 , piece_id=N7TEHBR7IWNLBUILH332AC7WNSJNZ4XYDS7AX5GCECPCPI7FB4HA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.411471208 (323h29m55.873563681s) [noise=true , address=145.79.227.14:4839 , piece_id=LNOEB3X2XDTT47M6GATWJWVYA3R62L5XMXLZKEJ374GKPUM4KTRQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.412301959 (323h29m55.874394442s) [noise=true , address=145.79.227.19:4077 , piece_id=CLZNAOMAJXXZLVODTRFFOL52LUO5BQQJL224AMBPHGWT4AMSME3A, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413077777 (323h29m55.87517025s) [noise=true , address=45.91.116.5:4299 , piece_id=TQ5EOVF2WRBCJACPJM3737IOCFAR6NAYWDYIVUFZY6U2YU3JO2SA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413115959 (323h29m55.875208432s) [noise=true , address=77.90.31.14:4663 , piece_id=HMMJUK3GQH73MREUCXLUQJBHTZAOADUJCVWFCR3AJBTCP4GRO5AQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413422523 (323h29m55.875515006s) [noise=true , address=77.90.31.16:4733 , piece_id=CKP2NN7B7J2PAGORMCQ42MGISFEQQUQLIQR32CVD5YBVLVC2L4YA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413833645 (323h29m55.875926128s) [noise=true , address=77.90.31.18:4806 , piece_id=E7VGWTIYKDDKBRKWUGAINRUEJ47EBJYKEGS5X5HNOCKHZT5ZR45A, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413966208 (323h29m55.876058691s) [noise=true , address=145.79.227.17:4972 , piece_id=GTLUNX4O7C5EEIUP25RZJWGQ2OBTHOBBZDWM76QAQMCRLTRR2N2Q, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.414258555 (323h29m55.876351028s) [noise=true , address=145.79.227.7:4293 , piece_id=OQQFJVQBPFQPMKCJOJLPK3VWT5ZQRZCFNOGPWOGX4ZD7BXKM6GHA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.414710566 (323h29m55.876803049s) [noise=true , address=77.90.31.11:4553 , piece_id=3NDPW2SUKFOGTUHP7C2OOICBT6DND77SHIN6GX3SBUOJCQKKHNMA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.414886451 (323h29m55.876978924s) [noise=true , address=145.79.227.15:4873 , piece_id=RALEQ5GZAQ4UFZUKOCQZLLWOJBXWXP5A2RDQHNCO226NK5WFAYQA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.415062666 (323h29m55.877155149s) [noise=true , address=145.79.227.3:4150 , piece_id=KCZXJ26XHDVDXR7R5NVZKJEASLQDH4ASUZE2CYHSJWW7MN3J5LHQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.415419926 (323h29m55.877512409s) [noise=true , address=145.79.227.13:4803 , piece_id=7KCTQU6ZBG6FZQHSQN64PXJUXDNCHUCDQRFUNCC5NNJL3MF4FQSQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.415697455 (323h29m55.877789948s) [noise=true , address=145.79.227.19:4078 , piece_id=5CBQOJN4YUTUONPOXK2TNPNYLBEYPBJJW3UFZ4Z55K2TFJ6L2NCA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.416700565 (323h29m55.878793038s) [noise=true , address=77.90.31.19:4842 , piece_id=NNKDBWZPJJEB7SLFEBUP5LO4HBHQ7MSPPUHV6CQIDRIHJV6OGS6A, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.417369609 (323h29m55.879462092s) [noise=true , address=45.91.116.12:4226 , piece_id=V63GMYNF6RXLUHYPLSYX4CGQF4BBPF7IOFVXSFT4N77T7RCGY4LA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:12.119016842 (323h29m57.581109325s) [upload=284702]: Next returned nil segment Oct 7 01:01:12.119114698 (323h29m57.581207201s) [upload=284702]: Waiting for error group managing segments... Oct 7 01:01:12.398844259 (323h29m57.860936742s) [noise=true , address=145.79.227.19:4077 , piece_id=CLZNAOMAJXXZLVODTRFFOL52LUO5BQQJL224AMBPHGWT4AMSME3A, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.402151106 (323h29m57.864243579s) [noise=true , address=145.79.227.14:4839 , piece_id=LNOEB3X2XDTT47M6GATWJWVYA3R62L5XMXLZKEJ374GKPUM4KTRQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.405187528 (323h29m57.867280011s) [noise=true , address=145.79.227.3:4150 , piece_id=KCZXJ26XHDVDXR7R5NVZKJEASLQDH4ASUZE2CYHSJWW7MN3J5LHQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.421155123 (323h29m57.883247596s) [noise=true , address=145.79.227.15:4873 , piece_id=RALEQ5GZAQ4UFZUKOCQZLLWOJBXWXP5A2RDQHNCO226NK5WFAYQA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.421662399 (323h29m57.883754883s) [noise=true , address=145.79.227.13:4803 , piece_id=7KCTQU6ZBG6FZQHSQN64PXJUXDNCHUCDQRFUNCC5NNJL3MF4FQSQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.423239743 (323h29m57.885332226s) [noise=true , address=145.79.227.17:4972 , piece_id=GTLUNX4O7C5EEIUP25RZJWGQ2OBTHOBBZDWM76QAQMCRLTRR2N2Q, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.425523362 (323h29m57.887615835s) [noise=true , address=145.79.227.19:4078 , piece_id=5CBQOJN4YUTUONPOXK2TNPNYLBEYPBJJW3UFZ4Z55K2TFJ6L2NCA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.425952498 (323h29m57.888044971s) [noise=true , address=145.79.227.4:4186 , piece_id=N7TEHBR7IWNLBUILH332AC7WNSJNZ4XYDS7AX5GCECPCPI7FB4HA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.431251198 (323h29m57.893343681s) [noise=true , address=145.79.227.7:4293 , piece_id=OQQFJVQBPFQPMKCJOJLPK3VWT5ZQRZCFNOGPWOGX4ZD7BXKM6GHA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.636922390 (323h29m58.099014873s) [noise=true , address=77.90.31.11:4553 , piece_id=3NDPW2SUKFOGTUHP7C2OOICBT6DND77SHIN6GX3SBUOJCQKKHNMA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.683157060 (323h29m58.145249543s) [noise=true , address=77.90.31.19:4842 , piece_id=NNKDBWZPJJEB7SLFEBUP5LO4HBHQ7MSPPUHV6CQIDRIHJV6OGS6A, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.683342783 (323h29m58.145435256s) [noise=true , address=45.91.116.5:4299 , piece_id=TQ5EOVF2WRBCJACPJM3737IOCFAR6NAYWDYIVUFZY6U2YU3JO2SA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.709183959 (323h29m58.171276442s) [noise=true , address=77.90.31.14:4663 , piece_id=HMMJUK3GQH73MREUCXLUQJBHTZAOADUJCVWFCR3AJBTCP4GRO5AQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.709211742 (323h29m58.171304235s) [seg_pos={0 0}, upload=284702]: Segment reached optimal threshold of 13 pieces. Oct 7 01:01:12.709372308 (323h29m58.171464791s) [noise=true , address=77.90.31.18:4806 , piece_id=E7VGWTIYKDDKBRKWUGAINRUEJ47EBJYKEGS5X5HNOCKHZT5ZR45A, seg_pos={0 0}, upload=284702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:38918->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:38918->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38918->77.90.31.18:4806: use of closed network connection Oct 7 01:01:12.709384090 (323h29m58.171476573s) [noise=true , address=77.90.31.18:4806 , piece_id=E7VGWTIYKDDKBRKWUGAINRUEJ47EBJYKEGS5X5HNOCKHZT5ZR45A, seg_pos={0 0}, upload=284702]: Long tail cancellation due to optimal threshold Oct 7 01:01:12.709527122 (323h29m58.171619595s) [noise=true , address=45.91.116.12:4226 , piece_id=V63GMYNF6RXLUHYPLSYX4CGQF4BBPF7IOFVXSFT4N77T7RCGY4LA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 7 01:01:12.709535739 (323h29m58.171628222s) [noise=true , address=45.91.116.12:4226 , piece_id=V63GMYNF6RXLUHYPLSYX4CGQF4BBPF7IOFVXSFT4N77T7RCGY4LA, seg_pos={0 0}, upload=284702]: Long tail cancellation due to optimal threshold Oct 7 01:01:12.709614038 (323h29m58.171706521s) [noise=true , address=77.90.31.16:4733 , piece_id=CKP2NN7B7J2PAGORMCQ42MGISFEQQUQLIQR32CVD5YBVLVC2L4YA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 7 01:01:12.709624408 (323h29m58.171716891s) [noise=true , address=77.90.31.16:4733 , piece_id=CKP2NN7B7J2PAGORMCQ42MGISFEQQUQLIQR32CVD5YBVLVC2L4YA, seg_pos={0 0}, upload=284702]: Long tail cancellation due to optimal threshold Oct 7 01:01:12.709639517 (323h29m58.171732s) [seg_pos={0 0}, upload=284702]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:01:12.709732493 (323h29m58.171824976s) [upload=284702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:01:12.709738976 (323h29m58.171831459s) [upload=284702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:01:13.360148075 (323h29m58.822240558s) [upload=284702]: Done uploading segments... Oct 7 01:01:13.360170197 (323h29m58.82226267s) [upload=284702]: Upload finished. err: <nil> Duration: 30.002422266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 31.960621015s FAIL (2025-10-07 01:01:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 31.960654528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................X.XXX...X..........] duration: 21.496968072s FAIL (2025-10-07 01:01:33 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 21.49699802s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................................X.XXX...X.........] duration: 17.563188905s FAIL (2025-10-07 01:01:09 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 17.563214292s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................X..XX..XX...X..] duration: 9.164958742s FAIL (2025-10-07 01:00:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q) context deadline exceeded; piecestore: (Node ID: 12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, Piece ID: HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ) context deadline exceeded ======LOGS====== Oct 7 00:59:12.685186317 (323h27m58.14727879s) [upload=284675]: Starting upload Oct 7 00:59:12.685278293 (323h27m58.147370776s) [upload=284675]: Done starting upload Oct 7 00:59:12.685454518 (323h27m58.147547001s) [upload=284675]: Uploading segments... Oct 7 00:59:12.685581469 (323h27m58.147673942s) [upload=284675]: Got next segment. Inline: false Oct 7 00:59:12.685608781 (323h27m58.147701265s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:59:12.685613420 (323h27m58.147705903s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:59:13.016521515 (323h27m58.478613998s) [seg_pos={0 0}, upload=284675]: Begin upload segment... Oct 7 00:59:13.016598392 (323h27m58.478690875s) [seg_pos={0 0}, upload=284675]: Done begin upload segment. Oct 7 00:59:13.016636374 (323h27m58.478728857s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016653387 (323h27m58.47874586s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016711817 (323h27m58.4788043s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016777923 (323h27m58.478870406s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017181683 (323h27m58.479274166s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017556496 (323h27m58.47964897s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017921681 (323h27m58.480014164s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018279232 (323h27m58.480371715s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018971991 (323h27m58.481064474s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019174026 (323h27m58.481266499s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019858951 (323h27m58.481951424s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020800053 (323h27m58.482892536s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020990314 (323h27m58.483082807s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.022234434 (323h27m58.484326917s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.025144545 (323h27m58.487237029s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.027054692 (323h27m58.489147175s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:14.718509397 (323h28m0.18060188s) [upload=284675]: Next returned nil segment Oct 7 00:59:14.718532982 (323h28m0.180625465s) [upload=284675]: Waiting for error group managing segments... Oct 7 00:59:14.997286224 (323h28m0.459378697s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.005665701 (323h28m0.467758194s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.012184233 (323h28m0.474276716s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015080098 (323h28m0.477172581s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015967287 (323h28m0.47805976s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.017137155 (323h28m0.479229628s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.033318026 (323h28m0.495410509s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.057300663 (323h28m0.519393136s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.174223514 (323h28m0.636315997s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247155398 (323h28m0.709247871s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247337565 (323h28m0.709430048s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.314344858 (323h28m0.776437341s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323081303 (323h28m0.785173787s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323112513 (323h28m0.785204996s) [seg_pos={0 0}, upload=284675]: Segment reached optimal threshold of 13 pieces. Oct 7 00:59:15.323280172 (323h28m0.785372655s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323294269 (323h28m0.785386742s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323360255 (323h28m0.785452728s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323367799 (323h28m0.785460282s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323548955 (323h28m0.785641438s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323620661 (323h28m0.785713144s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323725931 (323h28m0.785818414s) [seg_pos={0 0}, upload=284675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:59:15.323845278 (323h28m0.785937751s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:59:15.323856119 (323h28m0.785948602s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:59:15.677146474 (323h28m1.139238947s) [upload=284675]: Done uploading segments... Oct 7 00:59:15.677173235 (323h28m1.139265718s) [upload=284675]: Upload finished. err: <nil> Duration: 9.16498928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................X.XXX...X........] duration: 9.488944926s FAIL (2025-10-07 01:00:31 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 9.488973642s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.085871393s FAIL (2025-10-07 01:00:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 6.085904757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................X..XX..XX...X.] duration: 6.210321006s FAIL (2025-10-07 01:00:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q) context deadline exceeded; piecestore: (Node ID: 12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, Piece ID: HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ) context deadline exceeded ======LOGS====== Oct 7 00:59:12.685186317 (323h27m58.14727879s) [upload=284675]: Starting upload Oct 7 00:59:12.685278293 (323h27m58.147370776s) [upload=284675]: Done starting upload Oct 7 00:59:12.685454518 (323h27m58.147547001s) [upload=284675]: Uploading segments... Oct 7 00:59:12.685581469 (323h27m58.147673942s) [upload=284675]: Got next segment. Inline: false Oct 7 00:59:12.685608781 (323h27m58.147701265s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:59:12.685613420 (323h27m58.147705903s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:59:13.016521515 (323h27m58.478613998s) [seg_pos={0 0}, upload=284675]: Begin upload segment... Oct 7 00:59:13.016598392 (323h27m58.478690875s) [seg_pos={0 0}, upload=284675]: Done begin upload segment. Oct 7 00:59:13.016636374 (323h27m58.478728857s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016653387 (323h27m58.47874586s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016711817 (323h27m58.4788043s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016777923 (323h27m58.478870406s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017181683 (323h27m58.479274166s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017556496 (323h27m58.47964897s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017921681 (323h27m58.480014164s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018279232 (323h27m58.480371715s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018971991 (323h27m58.481064474s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019174026 (323h27m58.481266499s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019858951 (323h27m58.481951424s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020800053 (323h27m58.482892536s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020990314 (323h27m58.483082807s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.022234434 (323h27m58.484326917s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.025144545 (323h27m58.487237029s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.027054692 (323h27m58.489147175s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:14.718509397 (323h28m0.18060188s) [upload=284675]: Next returned nil segment Oct 7 00:59:14.718532982 (323h28m0.180625465s) [upload=284675]: Waiting for error group managing segments... Oct 7 00:59:14.997286224 (323h28m0.459378697s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.005665701 (323h28m0.467758194s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.012184233 (323h28m0.474276716s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015080098 (323h28m0.477172581s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015967287 (323h28m0.47805976s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.017137155 (323h28m0.479229628s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.033318026 (323h28m0.495410509s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.057300663 (323h28m0.519393136s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.174223514 (323h28m0.636315997s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247155398 (323h28m0.709247871s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247337565 (323h28m0.709430048s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.314344858 (323h28m0.776437341s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323081303 (323h28m0.785173787s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323112513 (323h28m0.785204996s) [seg_pos={0 0}, upload=284675]: Segment reached optimal threshold of 13 pieces. Oct 7 00:59:15.323280172 (323h28m0.785372655s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323294269 (323h28m0.785386742s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323360255 (323h28m0.785452728s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323367799 (323h28m0.785460282s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323548955 (323h28m0.785641438s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323620661 (323h28m0.785713144s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323725931 (323h28m0.785818414s) [seg_pos={0 0}, upload=284675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:59:15.323845278 (323h28m0.785937751s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:59:15.323856119 (323h28m0.785948602s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:59:15.677146474 (323h28m1.139238947s) [upload=284675]: Done uploading segments... Oct 7 00:59:15.677173235 (323h28m1.139265718s) [upload=284675]: Upload finished. err: <nil> Duration: 6.210379197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................................X.XXX...X.......] duration: 6.700427431s FAIL (2025-10-07 01:00:10 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.70047349s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................X.XXX...X......] duration: 18.12080563s FAIL (2025-10-07 00:59:52 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 18.120835577s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................................X..XX..XX...X] duration: 30.002479192s FAIL (2025-10-07 00:59:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q) context deadline exceeded; piecestore: (Node ID: 12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, Piece ID: HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ) context deadline exceeded ======LOGS====== Oct 7 00:59:12.685186317 (323h27m58.14727879s) [upload=284675]: Starting upload Oct 7 00:59:12.685278293 (323h27m58.147370776s) [upload=284675]: Done starting upload Oct 7 00:59:12.685454518 (323h27m58.147547001s) [upload=284675]: Uploading segments... Oct 7 00:59:12.685581469 (323h27m58.147673942s) [upload=284675]: Got next segment. Inline: false Oct 7 00:59:12.685608781 (323h27m58.147701265s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:59:12.685613420 (323h27m58.147705903s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:59:13.016521515 (323h27m58.478613998s) [seg_pos={0 0}, upload=284675]: Begin upload segment... Oct 7 00:59:13.016598392 (323h27m58.478690875s) [seg_pos={0 0}, upload=284675]: Done begin upload segment. Oct 7 00:59:13.016636374 (323h27m58.478728857s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016653387 (323h27m58.47874586s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016711817 (323h27m58.4788043s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016777923 (323h27m58.478870406s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017181683 (323h27m58.479274166s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017556496 (323h27m58.47964897s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017921681 (323h27m58.480014164s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018279232 (323h27m58.480371715s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018971991 (323h27m58.481064474s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019174026 (323h27m58.481266499s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019858951 (323h27m58.481951424s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020800053 (323h27m58.482892536s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020990314 (323h27m58.483082807s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.022234434 (323h27m58.484326917s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.025144545 (323h27m58.487237029s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.027054692 (323h27m58.489147175s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:14.718509397 (323h28m0.18060188s) [upload=284675]: Next returned nil segment Oct 7 00:59:14.718532982 (323h28m0.180625465s) [upload=284675]: Waiting for error group managing segments... Oct 7 00:59:14.997286224 (323h28m0.459378697s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.005665701 (323h28m0.467758194s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.012184233 (323h28m0.474276716s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015080098 (323h28m0.477172581s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015967287 (323h28m0.47805976s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.017137155 (323h28m0.479229628s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.033318026 (323h28m0.495410509s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.057300663 (323h28m0.519393136s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.174223514 (323h28m0.636315997s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247155398 (323h28m0.709247871s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247337565 (323h28m0.709430048s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.314344858 (323h28m0.776437341s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323081303 (323h28m0.785173787s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323112513 (323h28m0.785204996s) [seg_pos={0 0}, upload=284675]: Segment reached optimal threshold of 13 pieces. Oct 7 00:59:15.323280172 (323h28m0.785372655s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323294269 (323h28m0.785386742s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323360255 (323h28m0.785452728s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323367799 (323h28m0.785460282s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323548955 (323h28m0.785641438s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323620661 (323h28m0.785713144s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323725931 (323h28m0.785818414s) [seg_pos={0 0}, upload=284675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:59:15.323845278 (323h28m0.785937751s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:59:15.323856119 (323h28m0.785948602s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:59:15.677146474 (323h28m1.139238947s) [upload=284675]: Done uploading segments... Oct 7 00:59:15.677173235 (323h28m1.139265718s) [upload=284675]: Upload finished. err: <nil> Duration: 30.002560558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 28.852701972s FAIL (2025-10-07 00:59:34 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 28.852733713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................................X.XXX...X.....] duration: 8.106485594s FAIL (2025-10-07 00:58:37 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 8.10654163s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................X..XX..XX...] duration: 6.121925497s FAIL (2025-10-07 00:58:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA) context deadline exceeded ======LOGS====== t 7 00:56:00.189858132 (323h24m45.651950605s) [upload=284627]: Starting upload Oct 7 00:56:00.189920731 (323h24m45.652013214s) [upload=284627]: Done starting upload Oct 7 00:56:00.190097077 (323h24m45.65218956s) [upload=284627]: Uploading segments... Oct 7 00:56:00.190132274 (323h24m45.652224747s) [upload=284627]: Got next segment. Inline: false Oct 7 00:56:00.190160387 (323h24m45.65225286s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:56:00.190164555 (323h24m45.652257028s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:56:00.709885871 (323h24m46.171978354s) [seg_pos={0 0}, upload=284627]: Begin upload segment... Oct 7 00:56:00.709967937 (323h24m46.17206041s) [seg_pos={0 0}, upload=284627]: Done begin upload segment. Oct 7 00:56:00.710011539 (323h24m46.172104022s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710063549 (323h24m46.172156022s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710559473 (323h24m46.172651946s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710846559 (323h24m46.172939032s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711109100 (323h24m46.173201573s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711369365 (323h24m46.173461838s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711696338 (323h24m46.173788811s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711973806 (323h24m46.174066279s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.712251736 (323h24m46.174344209s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.714188002 (323h24m46.176280475s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.715636901 (323h24m46.177729384s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.719018861 (323h24m46.181111344s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.722245726 (323h24m46.184338209s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.725012714 (323h24m46.187105197s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.726486110 (323h24m46.188578584s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.730841143 (323h24m46.192933626s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:02.423096965 (323h24m47.885189448s) [upload=284627]: Next returned nil segment Oct 7 00:56:02.423122163 (323h24m47.885214636s) [upload=284627]: Waiting for error group managing segments... Oct 7 00:56:02.716933639 (323h24m48.179026122s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.719843060 (323h24m48.181935543s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.720087635 (323h24m48.182180118s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.727934226 (323h24m48.190026709s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733271680 (323h24m48.195364153s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733506647 (323h24m48.19559914s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.735797249 (323h24m48.197889742s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.743416728 (323h24m48.205509211s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.926212611 (323h24m48.388305085s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.956742295 (323h24m48.418834768s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.961749390 (323h24m48.423841873s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.040720374 (323h24m48.502812857s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043108311 (323h24m48.505200784s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043133209 (323h24m48.505225692s) [seg_pos={0 0}, upload=284627]: Segment reached optimal threshold of 13 pieces. Oct 7 00:56:03.043283535 (323h24m48.505376018s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043297180 (323h24m48.505389653s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043544391 (323h24m48.505636874s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection Oct 7 00:56:03.043555633 (323h24m48.505648116s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043658399 (323h24m48.505750882s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043667406 (323h24m48.505759889s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043684769 (323h24m48.505777252s) [seg_pos={0 0}, upload=284627]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:56:03.043766575 (323h24m48.505859058s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:56:03.043773057 (323h24m48.50586554s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:56:05.885462471 (323h24m51.347554964s) [upload=284627]: Done uploading segments... Oct 7 00:56:05.885488640 (323h24m51.347581124s) [upload=284627]: Upload finished. err: <nil> Duration: 6.121957428s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.986583783s FAIL (2025-10-07 00:58:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.986623378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.94838116s FAIL (2025-10-07 00:58:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.948415185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................X.XXX...X....] duration: 5.976962933s FAIL (2025-10-07 00:57:56 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.976993792s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................X..XX..XX..] duration: 6.085000873s FAIL (2025-10-07 00:57:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA) context deadline exceeded ======LOGS====== t 7 00:56:00.189858132 (323h24m45.651950605s) [upload=284627]: Starting upload Oct 7 00:56:00.189920731 (323h24m45.652013214s) [upload=284627]: Done starting upload Oct 7 00:56:00.190097077 (323h24m45.65218956s) [upload=284627]: Uploading segments... Oct 7 00:56:00.190132274 (323h24m45.652224747s) [upload=284627]: Got next segment. Inline: false Oct 7 00:56:00.190160387 (323h24m45.65225286s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:56:00.190164555 (323h24m45.652257028s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:56:00.709885871 (323h24m46.171978354s) [seg_pos={0 0}, upload=284627]: Begin upload segment... Oct 7 00:56:00.709967937 (323h24m46.17206041s) [seg_pos={0 0}, upload=284627]: Done begin upload segment. Oct 7 00:56:00.710011539 (323h24m46.172104022s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710063549 (323h24m46.172156022s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710559473 (323h24m46.172651946s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710846559 (323h24m46.172939032s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711109100 (323h24m46.173201573s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711369365 (323h24m46.173461838s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711696338 (323h24m46.173788811s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711973806 (323h24m46.174066279s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.712251736 (323h24m46.174344209s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.714188002 (323h24m46.176280475s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.715636901 (323h24m46.177729384s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.719018861 (323h24m46.181111344s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.722245726 (323h24m46.184338209s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.725012714 (323h24m46.187105197s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.726486110 (323h24m46.188578584s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.730841143 (323h24m46.192933626s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:02.423096965 (323h24m47.885189448s) [upload=284627]: Next returned nil segment Oct 7 00:56:02.423122163 (323h24m47.885214636s) [upload=284627]: Waiting for error group managing segments... Oct 7 00:56:02.716933639 (323h24m48.179026122s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.719843060 (323h24m48.181935543s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.720087635 (323h24m48.182180118s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.727934226 (323h24m48.190026709s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733271680 (323h24m48.195364153s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733506647 (323h24m48.19559914s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.735797249 (323h24m48.197889742s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.743416728 (323h24m48.205509211s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.926212611 (323h24m48.388305085s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.956742295 (323h24m48.418834768s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.961749390 (323h24m48.423841873s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.040720374 (323h24m48.502812857s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043108311 (323h24m48.505200784s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043133209 (323h24m48.505225692s) [seg_pos={0 0}, upload=284627]: Segment reached optimal threshold of 13 pieces. Oct 7 00:56:03.043283535 (323h24m48.505376018s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043297180 (323h24m48.505389653s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043544391 (323h24m48.505636874s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection Oct 7 00:56:03.043555633 (323h24m48.505648116s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043658399 (323h24m48.505750882s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043667406 (323h24m48.505759889s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043684769 (323h24m48.505777252s) [seg_pos={0 0}, upload=284627]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:56:03.043766575 (323h24m48.505859058s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:56:03.043773057 (323h24m48.50586554s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:56:05.885462471 (323h24m51.347554964s) [upload=284627]: Done uploading segments... Oct 7 00:56:05.885488640 (323h24m51.347581124s) [upload=284627]: Upload finished. err: <nil> Duration: 6.085032894s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)