Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.96278009s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:05:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.962807501s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................................XX......] duration: 3.421466974s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:05:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42149558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.7919483s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:05:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.791974819s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.155630885s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:05:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.155663606s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.90463407s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:05:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.904768267s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................................XX.....] duration: 3.563218361s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:05:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563409546s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.186369696s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:04:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.186536745s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................................XX....] duration: 3.519163214s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:04:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.519312208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.856261643s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:04:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.856293804s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.222220987s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:03:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.222247929s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.247339974s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:03:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.247368719s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................................XX...] duration: 3.497695565s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:03:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497752582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.853482759s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:03:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.853522195s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.748801418s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:02:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.74883348s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.18701873s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:02:23 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.187046914s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................................XX..] duration: 3.51390463s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:02:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513933464s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.216639644s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:01:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.216661786s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.745932097s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:01:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.745958457s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.129488615s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:01:17 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.129520957s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................................XX.] duration: 3.545664035s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:01:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.545688513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.825774628s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:01:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.825804536s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.144423894s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:01:02 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.144475774s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.791854759s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:00:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.791879136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.251269602s trace: https://jaeger.dev.storj.io/trace/39fc781377eb8ee5 FAIL (2025-11-25 03:00:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 03:00:12.188411433 (378h56m10.739881894s) [upload=333709]: Starting upload Nov 25 03:00:12.188530148 (378h56m10.740000619s) [upload=333709]: Done starting upload Nov 25 03:00:12.188645808 (378h56m10.740116299s) [upload=333709]: Uploading segments... Nov 25 03:00:12.189261170 (378h56m10.740731631s) [upload=333709]: Got next segment. Inline: false Nov 25 03:00:12.189408000 (378h56m10.740878471s) [upload=333709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 03:00:12.189415955 (378h56m10.740886417s) [upload=333709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 03:00:12.436858145 (378h56m10.988328616s) [seg_pos={0 0}, upload=333709]: Begin upload segment... Nov 25 03:00:12.437032056 (378h56m10.988502517s) [seg_pos={0 0}, upload=333709]: Done begin upload segment. Nov 25 03:00:12.437072484 (378h56m10.988542945s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439374517 (378h56m10.990845008s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.439406649 (378h56m10.99087712s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440062947 (378h56m10.991533408s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440664524 (378h56m10.992134995s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.440752200 (378h56m10.992222661s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.441416044 (378h56m10.992886505s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.442280310 (378h56m10.993750771s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.443633717 (378h56m10.995104178s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444389376 (378h56m10.995859837s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444416026 (378h56m10.995886487s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.444483204 (378h56m10.995953675s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.445028713 (378h56m10.996499174s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446040891 (378h56m10.997511352s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.446587622 (378h56m10.998058093s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:12.448141290 (378h56m10.999611751s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Uploading piece... Nov 25 03:00:14.252252192 (378h56m12.803722663s) [upload=333709]: Next returned nil segment Nov 25 03:00:14.252274915 (378h56m12.803745376s) [upload=333709]: Waiting for error group managing segments... Nov 25 03:00:14.680043185 (378h56m13.231513646s) [noise=true , address=77.90.31.14:4663 , piece_id=XSGJOTWWQGZJ3U42T7O7BIUCF753NH3TT6V3YGU5TKVX3K7KERIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.695301909 (378h56m13.24677237s) [noise=true , address=77.90.31.19:4843 , piece_id=USMKZKVFUEAU3KSYF4GHJWDOLD3U7QBVZQPKXUEGC47MF5HY4SQA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.696179109 (378h56m13.24764957s) [noise=true , address=45.91.116.9:4263 , piece_id=EGXQTVRDSG67ZF3EP6MUVWI7NC7RQHJBOXTDQNZUGLSK532K5DKQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.708674132 (378h56m13.260144583s) [noise=true , address=77.90.31.13:4628 , piece_id=XLGLOCVUWOER6WSSP6UGHDZEEMLSNXYQK7HH3YS3WK7LGEA3IA6Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:14.768497536 (378h56m13.319967998s) [noise=true , address=77.90.31.19:4842 , piece_id=64FJIZ6FELOM35DB3K364FOXN3N47LQNVI55NEGNC2RW3LJJKONQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.638558740 (378h56m35.190029211s) [noise=true , address=145.79.227.26:4653 , piece_id=ZTGMSOMN2FOR4V27MWG375NPFDMCR3YUZL4S5GQXGLAQDOS2PH2A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:36.822018967 (378h56m35.373489429s) [noise=true , address=145.79.227.27:4691 , piece_id=TAYD247CU2QTBCVR4EITWZP33SLJD4FADSSTLPEWRWMBNHXQ2GDQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.048515463 (378h56m35.599985924s) [noise=true , address=145.79.227.13:4802 , piece_id=KQYNNTE5H6UO3VGL7YCRNLJY2LA75INJRZUMV366QRTGTY6OOCGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.673647937 (378h56m36.225118399s) [noise=true , address=145.79.227.40:4051 , piece_id=GC6K66SGRZBJBBDFBGK5AUFVQQ27Q7XVRR5CWQZC3PL2YWIOOHGQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:37.978354408 (378h56m36.529824869s) [noise=true , address=145.79.227.28:4726 , piece_id=LC6QQOBGEWHVBKFMF6CSCR43RDIN3OYW2X6WH2R6ITQFKCUBJMIA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:40.197034208 (378h56m38.748504669s) [noise=true , address=145.79.227.20:4437 , piece_id=JP2HKINYTDRZXXSWXO2K6OJWE2HIBCHKBMKDKDIM2KSATVK7MDVQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.205827806 (378h56m39.757298267s) [noise=true , address=145.79.227.29:4943 , piece_id=BREYGDX6ZUQJZI66BDZ4DL5KK7TA76TXHHEUJ2CFLHWR5DLA3ATA, seg_pos={0 0}, upload=333709]: Done uploading piece. err: <nil> Nov 25 03:00:41.487902312 (378h56m40.039372783s) [noise=true , address=145.79.227.24:4583 , piece_id=KEFP55AR6IH3HRKBYVQQ2BRPLDO3KGMTF3KLNP4XV4GNHYRJBZBQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: rpc: dial tcp 145.79.227.24:4583: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 03:00:41.488487326 (378h56m40.039957787s) [noise=true , address=145.79.227.44:4194 , piece_id=Y4Z5GTF54JYCEDUUF6AS2JQRXIWNIHYNURKITEBIO52MEK2IWKIQ, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40048->145.79.227.44:4194: use of closed network connection Nov 25 03:00:41.488597636 (378h56m40.040068107s) [noise=true , address=145.79.227.15:4876 , piece_id=PQX6Q3Y6GDRG2KLHQPX5WZ77PWO5PSKKYJM73KWEEJOSOCGMOR6A, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60216->145.79.227.15:4876: use of closed network connection Nov 25 03:00:41.488704790 (378h56m40.040175261s) [noise=true , address=145.79.227.10:4403 , piece_id=VPE7QBVH66YGIZH2KTRMDQV2CEBABCHKYA4X5M6V736TUL4CN75Q, seg_pos={0 0}, upload=333709]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58542->145.79.227.10:4403: use of closed network connection Nov 25 03:00:41.488747161 (378h56m40.040217622s) [seg_pos={0 0}, upload=333709]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 03:00:41.735489939 (378h56m40.28696041s) [upload=333709]: Done uploading segments... Nov 25 03:00:41.735535386 (378h56m40.287005847s) [upload=333709]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.251298446s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................................XX] duration: 5.001478063s trace: https://jaeger.dev.storj.io/trace/186921e0a772147d FAIL (2025-11-25 03:00:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001502529s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.196853657s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 03:00:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.196880086s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.002303012s trace: https://jaeger.dev.storj.io/trace/22b85df98e1b4c57 FAIL (2025-11-25 02:59:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002330063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.183689944s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:59:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.183715852s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.140768192s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:58:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.140795484s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.235959042s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:56:56 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.235984591s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.237525356s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:56:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.237552727s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.163769592s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:55:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.163788968s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.162565659s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:54:50 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.162592861s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.153584943s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:54:06 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.153622564s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.197913242s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:53:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.198018412s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.17237325s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:52:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.172399088s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.174492955s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:51:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.174511139s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.198855801s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:50:49 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.198904092s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.131880765s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:50:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.13191464s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.159663417s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:49:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.159688123s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.232758728s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:48:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.232788796s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.23289591s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:48:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.232963529s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.183926167s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:47:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.183957145s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.142414653s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:46:06 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.142462604s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.217718922s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:45:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.21773891s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.199593691s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:44:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.199618928s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.131954866s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:43:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.131983231s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.163883665s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:42:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.163913371s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.189305558s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:42:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.189327832s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.511067149s trace: https://jaeger.dev.storj.io/trace/61e525a9999a7625 FAIL (2025-11-25 02:41:26 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52218->45.38.24.2:443: read: connection reset by peer Duration: 2.511137744s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)