Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 428229 reports (Page 8287 of 8565)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X..........................X......X..........] duration: 30.6211624s FAIL (2025-07-09 01:43:09 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 9 01:34:13.079791339 (376h23m47.850432674s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:34:13.539243416 (376h23m48.309884751s) [seg_pos={0 0}, upload=301130]: Begin upload segment... Jul 9 01:34:13.539291719 (376h23m48.309933054s) [seg_pos={0 0}, upload=301130]: Done begin upload segment. Jul 9 01:34:13.539316947 (376h23m48.309958272s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539357194 (376h23m48.309998529s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539656567 (376h23m48.310297902s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539917576 (376h23m48.310558901s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539962171 (376h23m48.310603496s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540198754 (376h23m48.310840089s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540273528 (376h23m48.310914863s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540441268 (376h23m48.311082603s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540606304 (376h23m48.311247639s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540682078 (376h23m48.311323413s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540939712 (376h23m48.311581047s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540945533 (376h23m48.311586868s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541138272 (376h23m48.311779607s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541241369 (376h23m48.311882704s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541327143 (376h23m48.311968478s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541504031 (376h23m48.312145366s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541533798 (376h23m48.312175133s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541792674 (376h23m48.312433999s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:15.115895415 (376h23m49.88653675s) [upload=301130]: Next returned nil segment Jul 9 01:34:15.115912727 (376h23m49.886554062s) [upload=301130]: Waiting for error group managing segments... Jul 9 01:34:15.520996542 (376h23m50.291637867s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.527850836 (376h23m50.298492171s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529225055 (376h23m50.29986639s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529574193 (376h23m50.300215528s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.532332300 (376h23m50.302973635s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.534764755 (376h23m50.30540608s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.537145048 (376h23m50.307786383s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.542288250 (376h23m50.312929585s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.543383766 (376h23m50.314025101s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.558874324 (376h23m50.329515649s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.565242998 (376h23m50.335884333s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:25.557055824 (376h24m0.327697149s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:51.372897374 (376h24m26.143538709s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024816982 (376h24m46.795458317s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024848702 (376h24m46.795490037s) [seg_pos={0 0}, upload=301130]: Segment reached optimal threshold of 14 pieces. Jul 9 01:35:12.024962870 (376h24m46.795604205s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.024969803 (376h24m46.795611138s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.025076478 (376h24m46.795717813s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK): protocol: expected piece hash; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; EOF Jul 9 01:35:12.025102417 (376h24m46.795743742s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3): protocol: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection Jul 9 01:35:12.025112908 (376h24m46.795754233s) [seg_pos={0 0}, upload=301130]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:35:12.025165398 (376h24m46.795806734s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:35:12.025169095 (376h24m46.79581043s) [upload=301130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:35:12.344489789 (376h24m47.115131124s) [upload=301130]: Done uploading segments... Jul 9 01:35:12.344513775 (376h24m47.115155141s) [upload=301130]: Upload finished. err: <nil> Duration:* 30.621187727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X......XX....X...X.X....X...X.X..X...X..X............X.....] duration: 28.044837883s FAIL (2025-07-09 01:43:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.044884924s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [............................X......XX.XX..X......XXX....XX..] duration: 54.6605349s FAIL (2025-07-09 01:42:57 UTC) orbiter warning: context deadline exceeded Duration: 54.660547054s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X........................X......................X.] duration: 5.739246705s FAIL (2025-07-09 01:42:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments... Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment... Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment. Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments... Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments... Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.739278965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X..........................X......X.........] duration: 35.335885776s FAIL (2025-07-09 01:42:23 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 9 01:34:13.079791339 (376h23m47.850432674s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:34:13.539243416 (376h23m48.309884751s) [seg_pos={0 0}, upload=301130]: Begin upload segment... Jul 9 01:34:13.539291719 (376h23m48.309933054s) [seg_pos={0 0}, upload=301130]: Done begin upload segment. Jul 9 01:34:13.539316947 (376h23m48.309958272s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539357194 (376h23m48.309998529s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539656567 (376h23m48.310297902s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539917576 (376h23m48.310558901s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539962171 (376h23m48.310603496s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540198754 (376h23m48.310840089s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540273528 (376h23m48.310914863s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540441268 (376h23m48.311082603s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540606304 (376h23m48.311247639s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540682078 (376h23m48.311323413s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540939712 (376h23m48.311581047s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540945533 (376h23m48.311586868s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541138272 (376h23m48.311779607s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541241369 (376h23m48.311882704s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541327143 (376h23m48.311968478s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541504031 (376h23m48.312145366s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541533798 (376h23m48.312175133s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541792674 (376h23m48.312433999s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:15.115895415 (376h23m49.88653675s) [upload=301130]: Next returned nil segment Jul 9 01:34:15.115912727 (376h23m49.886554062s) [upload=301130]: Waiting for error group managing segments... Jul 9 01:34:15.520996542 (376h23m50.291637867s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.527850836 (376h23m50.298492171s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529225055 (376h23m50.29986639s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529574193 (376h23m50.300215528s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.532332300 (376h23m50.302973635s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.534764755 (376h23m50.30540608s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.537145048 (376h23m50.307786383s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.542288250 (376h23m50.312929585s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.543383766 (376h23m50.314025101s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.558874324 (376h23m50.329515649s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.565242998 (376h23m50.335884333s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:25.557055824 (376h24m0.327697149s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:51.372897374 (376h24m26.143538709s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024816982 (376h24m46.795458317s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024848702 (376h24m46.795490037s) [seg_pos={0 0}, upload=301130]: Segment reached optimal threshold of 14 pieces. Jul 9 01:35:12.024962870 (376h24m46.795604205s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.024969803 (376h24m46.795611138s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.025076478 (376h24m46.795717813s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK): protocol: expected piece hash; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; EOF Jul 9 01:35:12.025102417 (376h24m46.795743742s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3): protocol: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection Jul 9 01:35:12.025112908 (376h24m46.795754233s) [seg_pos={0 0}, upload=301130]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:35:12.025165398 (376h24m46.795806734s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:35:12.025169095 (376h24m46.79581043s) [upload=301130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:35:12.344489789 (376h24m47.115131124s) [upload=301130]: Done uploading segments... Jul 9 01:35:12.344513775 (376h24m47.115155141s) [upload=301130]: Upload finished. err: <nil> Duration:* 35.335911945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X........................X......................X] duration: 30.227299533s FAIL (2025-07-09 01:42:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments... Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment... Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment. Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece... Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments... Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil> Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments... Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.227317559s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.............................X......XX.XX..X......XXX....XX.] duration: 44.924350339s FAIL (2025-07-09 01:42:03 UTC) orbiter warning: context deadline exceeded Duration: 44.924362251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X......XX....X...X.X....X...X.X..X...X..X............X....] duration: 6.102758226s FAIL (2025-07-09 01:41:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.10278148s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X..........................X......X........] duration: 27.572702302s FAIL (2025-07-09 01:41:38 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 9 01:34:13.079791339 (376h23m47.850432674s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:34:13.539243416 (376h23m48.309884751s) [seg_pos={0 0}, upload=301130]: Begin upload segment... Jul 9 01:34:13.539291719 (376h23m48.309933054s) [seg_pos={0 0}, upload=301130]: Done begin upload segment. Jul 9 01:34:13.539316947 (376h23m48.309958272s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539357194 (376h23m48.309998529s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539656567 (376h23m48.310297902s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539917576 (376h23m48.310558901s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539962171 (376h23m48.310603496s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540198754 (376h23m48.310840089s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540273528 (376h23m48.310914863s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540441268 (376h23m48.311082603s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540606304 (376h23m48.311247639s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540682078 (376h23m48.311323413s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540939712 (376h23m48.311581047s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540945533 (376h23m48.311586868s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541138272 (376h23m48.311779607s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541241369 (376h23m48.311882704s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541327143 (376h23m48.311968478s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541504031 (376h23m48.312145366s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541533798 (376h23m48.312175133s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541792674 (376h23m48.312433999s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:15.115895415 (376h23m49.88653675s) [upload=301130]: Next returned nil segment Jul 9 01:34:15.115912727 (376h23m49.886554062s) [upload=301130]: Waiting for error group managing segments... Jul 9 01:34:15.520996542 (376h23m50.291637867s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.527850836 (376h23m50.298492171s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529225055 (376h23m50.29986639s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529574193 (376h23m50.300215528s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.532332300 (376h23m50.302973635s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.534764755 (376h23m50.30540608s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.537145048 (376h23m50.307786383s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.542288250 (376h23m50.312929585s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.543383766 (376h23m50.314025101s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.558874324 (376h23m50.329515649s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.565242998 (376h23m50.335884333s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:25.557055824 (376h24m0.327697149s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:51.372897374 (376h24m26.143538709s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024816982 (376h24m46.795458317s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024848702 (376h24m46.795490037s) [seg_pos={0 0}, upload=301130]: Segment reached optimal threshold of 14 pieces. Jul 9 01:35:12.024962870 (376h24m46.795604205s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.024969803 (376h24m46.795611138s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.025076478 (376h24m46.795717813s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK): protocol: expected piece hash; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; EOF Jul 9 01:35:12.025102417 (376h24m46.795743742s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3): protocol: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection Jul 9 01:35:12.025112908 (376h24m46.795754233s) [seg_pos={0 0}, upload=301130]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:35:12.025165398 (376h24m46.795806734s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:35:12.025169095 (376h24m46.79581043s) [upload=301130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:35:12.344489789 (376h24m47.115131124s) [upload=301130]: Done uploading segments... Jul 9 01:35:12.344513775 (376h24m47.115155141s) [upload=301130]: Upload finished. err: <nil> Duration:* 27.572727489s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..............................X......XX.XX..X......XXX....XX] duration: 1m0.000943658s FAIL (2025-07-09 01:41:17 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000956742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X........................X......................] duration: 5.751404643s FAIL (2025-07-09 01:41:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ommit Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments... Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil> Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments... Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment... Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment. Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments... Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments... Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.751433569s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X......XX....X...X.X....X...X.X..X...X..X............X...] duration: 12.757455089s FAIL (2025-07-09 01:40:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.757475609s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X..........................X......X.......] duration: 47.66706386s FAIL (2025-07-09 01:40:49 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 9 01:34:13.079791339 (376h23m47.850432674s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:34:13.539243416 (376h23m48.309884751s) [seg_pos={0 0}, upload=301130]: Begin upload segment... Jul 9 01:34:13.539291719 (376h23m48.309933054s) [seg_pos={0 0}, upload=301130]: Done begin upload segment. Jul 9 01:34:13.539316947 (376h23m48.309958272s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539357194 (376h23m48.309998529s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539656567 (376h23m48.310297902s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539917576 (376h23m48.310558901s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539962171 (376h23m48.310603496s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540198754 (376h23m48.310840089s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540273528 (376h23m48.310914863s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540441268 (376h23m48.311082603s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540606304 (376h23m48.311247639s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540682078 (376h23m48.311323413s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540939712 (376h23m48.311581047s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540945533 (376h23m48.311586868s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541138272 (376h23m48.311779607s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541241369 (376h23m48.311882704s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541327143 (376h23m48.311968478s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541504031 (376h23m48.312145366s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541533798 (376h23m48.312175133s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541792674 (376h23m48.312433999s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:15.115895415 (376h23m49.88653675s) [upload=301130]: Next returned nil segment Jul 9 01:34:15.115912727 (376h23m49.886554062s) [upload=301130]: Waiting for error group managing segments... Jul 9 01:34:15.520996542 (376h23m50.291637867s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.527850836 (376h23m50.298492171s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529225055 (376h23m50.29986639s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529574193 (376h23m50.300215528s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.532332300 (376h23m50.302973635s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.534764755 (376h23m50.30540608s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.537145048 (376h23m50.307786383s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.542288250 (376h23m50.312929585s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.543383766 (376h23m50.314025101s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.558874324 (376h23m50.329515649s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.565242998 (376h23m50.335884333s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:25.557055824 (376h24m0.327697149s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:51.372897374 (376h24m26.143538709s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024816982 (376h24m46.795458317s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024848702 (376h24m46.795490037s) [seg_pos={0 0}, upload=301130]: Segment reached optimal threshold of 14 pieces. Jul 9 01:35:12.024962870 (376h24m46.795604205s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.024969803 (376h24m46.795611138s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.025076478 (376h24m46.795717813s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK): protocol: expected piece hash; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; EOF Jul 9 01:35:12.025102417 (376h24m46.795743742s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3): protocol: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection Jul 9 01:35:12.025112908 (376h24m46.795754233s) [seg_pos={0 0}, upload=301130]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:35:12.025165398 (376h24m46.795806734s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:35:12.025169095 (376h24m46.79581043s) [upload=301130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:35:12.344489789 (376h24m47.115131124s) [upload=301130]: Done uploading segments... Jul 9 01:35:12.344513775 (376h24m47.115155141s) [upload=301130]: Upload finished. err: <nil> Duration:* 47.66709513s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...X......XX....X...X.X....X...X.X..X...X..X............X..] duration: 5.908411616s FAIL (2025-07-09 01:40:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.908448728s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [...............................X......XX.XX..X......XXX....X] duration: 1m0.000712135s FAIL (2025-07-09 01:40:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000724538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...X......XX....X...X.X....X...X.X..X...X..X............X.] duration: 6.464048698s FAIL (2025-07-09 01:40:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.464069738s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................X..........................X......X......] duration: 32.9477134s FAIL (2025-07-09 01:40:01 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 9 01:34:13.079791339 (376h23m47.850432674s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:34:13.539243416 (376h23m48.309884751s) [seg_pos={0 0}, upload=301130]: Begin upload segment... Jul 9 01:34:13.539291719 (376h23m48.309933054s) [seg_pos={0 0}, upload=301130]: Done begin upload segment. Jul 9 01:34:13.539316947 (376h23m48.309958272s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539357194 (376h23m48.309998529s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539656567 (376h23m48.310297902s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539917576 (376h23m48.310558901s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539962171 (376h23m48.310603496s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540198754 (376h23m48.310840089s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540273528 (376h23m48.310914863s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540441268 (376h23m48.311082603s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540606304 (376h23m48.311247639s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540682078 (376h23m48.311323413s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540939712 (376h23m48.311581047s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540945533 (376h23m48.311586868s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541138272 (376h23m48.311779607s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541241369 (376h23m48.311882704s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541327143 (376h23m48.311968478s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541504031 (376h23m48.312145366s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541533798 (376h23m48.312175133s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541792674 (376h23m48.312433999s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:15.115895415 (376h23m49.88653675s) [upload=301130]: Next returned nil segment Jul 9 01:34:15.115912727 (376h23m49.886554062s) [upload=301130]: Waiting for error group managing segments... Jul 9 01:34:15.520996542 (376h23m50.291637867s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.527850836 (376h23m50.298492171s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529225055 (376h23m50.29986639s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529574193 (376h23m50.300215528s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.532332300 (376h23m50.302973635s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.534764755 (376h23m50.30540608s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.537145048 (376h23m50.307786383s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.542288250 (376h23m50.312929585s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.543383766 (376h23m50.314025101s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.558874324 (376h23m50.329515649s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.565242998 (376h23m50.335884333s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:25.557055824 (376h24m0.327697149s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:51.372897374 (376h24m26.143538709s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024816982 (376h24m46.795458317s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024848702 (376h24m46.795490037s) [seg_pos={0 0}, upload=301130]: Segment reached optimal threshold of 14 pieces. Jul 9 01:35:12.024962870 (376h24m46.795604205s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.024969803 (376h24m46.795611138s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.025076478 (376h24m46.795717813s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK): protocol: expected piece hash; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; EOF Jul 9 01:35:12.025102417 (376h24m46.795743742s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3): protocol: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection Jul 9 01:35:12.025112908 (376h24m46.795754233s) [seg_pos={0 0}, upload=301130]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:35:12.025165398 (376h24m46.795806734s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:35:12.025169095 (376h24m46.79581043s) [upload=301130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:35:12.344489789 (376h24m47.115131124s) [upload=301130]: Done uploading segments... Jul 9 01:35:12.344513775 (376h24m47.115155141s) [upload=301130]: Upload finished. err: <nil> Duration:* 32.947743768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X........................X.....................] duration: 5.739959328s FAIL (2025-07-09 01:39:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ommit Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments... Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil> Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments... Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment... Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment. Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments... Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments... Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.739987192s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X...X......XX....X...X.X....X...X.X..X...X..X............X] duration: 30.001439843s FAIL (2025-07-09 01:39:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001463188s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [................................X......XX.XX..X......XXX....] duration: 48.705162386s FAIL (2025-07-09 01:39:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.705176213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................X..........................X......X.....] duration: 26.58904035s FAIL (2025-07-09 01:39:01 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 9 01:34:13.079791339 (376h23m47.850432674s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:34:13.539243416 (376h23m48.309884751s) [seg_pos={0 0}, upload=301130]: Begin upload segment... Jul 9 01:34:13.539291719 (376h23m48.309933054s) [seg_pos={0 0}, upload=301130]: Done begin upload segment. Jul 9 01:34:13.539316947 (376h23m48.309958272s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539357194 (376h23m48.309998529s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539656567 (376h23m48.310297902s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539917576 (376h23m48.310558901s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539962171 (376h23m48.310603496s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540198754 (376h23m48.310840089s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540273528 (376h23m48.310914863s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540441268 (376h23m48.311082603s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540606304 (376h23m48.311247639s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540682078 (376h23m48.311323413s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540939712 (376h23m48.311581047s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540945533 (376h23m48.311586868s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541138272 (376h23m48.311779607s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541241369 (376h23m48.311882704s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541327143 (376h23m48.311968478s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541504031 (376h23m48.312145366s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541533798 (376h23m48.312175133s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541792674 (376h23m48.312433999s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:15.115895415 (376h23m49.88653675s) [upload=301130]: Next returned nil segment Jul 9 01:34:15.115912727 (376h23m49.886554062s) [upload=301130]: Waiting for error group managing segments... Jul 9 01:34:15.520996542 (376h23m50.291637867s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.527850836 (376h23m50.298492171s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529225055 (376h23m50.29986639s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529574193 (376h23m50.300215528s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.532332300 (376h23m50.302973635s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.534764755 (376h23m50.30540608s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.537145048 (376h23m50.307786383s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.542288250 (376h23m50.312929585s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.543383766 (376h23m50.314025101s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.558874324 (376h23m50.329515649s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.565242998 (376h23m50.335884333s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:25.557055824 (376h24m0.327697149s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:51.372897374 (376h24m26.143538709s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024816982 (376h24m46.795458317s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024848702 (376h24m46.795490037s) [seg_pos={0 0}, upload=301130]: Segment reached optimal threshold of 14 pieces. Jul 9 01:35:12.024962870 (376h24m46.795604205s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.024969803 (376h24m46.795611138s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.025076478 (376h24m46.795717813s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK): protocol: expected piece hash; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; EOF Jul 9 01:35:12.025102417 (376h24m46.795743742s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3): protocol: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection Jul 9 01:35:12.025112908 (376h24m46.795754233s) [seg_pos={0 0}, upload=301130]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:35:12.025165398 (376h24m46.795806734s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:35:12.025169095 (376h24m46.79581043s) [upload=301130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:35:12.344489789 (376h24m47.115131124s) [upload=301130]: Done uploading segments... Jul 9 01:35:12.344513775 (376h24m47.115155141s) [upload=301130]: Upload finished. err: <nil> Duration:* 26.589056772s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X...X......XX....X...X.X....X...X.X..X...X..X............] duration: 24.979183059s FAIL (2025-07-09 01:38:51 UTC) orbiter warning: context deadline exceeded Duration: 24.979203607s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X........................X....................] duration: 5.737512817s FAIL (2025-07-09 01:38:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ommit Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments... Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil> Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments... Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment... Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment. Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments... Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments... Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.737543156s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..........................X......X....] duration: 40.085239138s FAIL (2025-07-09 01:38:34 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 9 01:34:13.079791339 (376h23m47.850432674s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:34:13.539243416 (376h23m48.309884751s) [seg_pos={0 0}, upload=301130]: Begin upload segment... Jul 9 01:34:13.539291719 (376h23m48.309933054s) [seg_pos={0 0}, upload=301130]: Done begin upload segment. Jul 9 01:34:13.539316947 (376h23m48.309958272s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539357194 (376h23m48.309998529s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539656567 (376h23m48.310297902s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539917576 (376h23m48.310558901s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539962171 (376h23m48.310603496s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540198754 (376h23m48.310840089s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540273528 (376h23m48.310914863s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540441268 (376h23m48.311082603s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540606304 (376h23m48.311247639s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540682078 (376h23m48.311323413s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540939712 (376h23m48.311581047s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540945533 (376h23m48.311586868s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541138272 (376h23m48.311779607s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541241369 (376h23m48.311882704s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541327143 (376h23m48.311968478s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541504031 (376h23m48.312145366s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541533798 (376h23m48.312175133s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541792674 (376h23m48.312433999s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:15.115895415 (376h23m49.88653675s) [upload=301130]: Next returned nil segment Jul 9 01:34:15.115912727 (376h23m49.886554062s) [upload=301130]: Waiting for error group managing segments... Jul 9 01:34:15.520996542 (376h23m50.291637867s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.527850836 (376h23m50.298492171s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529225055 (376h23m50.29986639s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529574193 (376h23m50.300215528s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.532332300 (376h23m50.302973635s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.534764755 (376h23m50.30540608s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.537145048 (376h23m50.307786383s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.542288250 (376h23m50.312929585s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.543383766 (376h23m50.314025101s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.558874324 (376h23m50.329515649s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.565242998 (376h23m50.335884333s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:25.557055824 (376h24m0.327697149s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:51.372897374 (376h24m26.143538709s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024816982 (376h24m46.795458317s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024848702 (376h24m46.795490037s) [seg_pos={0 0}, upload=301130]: Segment reached optimal threshold of 14 pieces. Jul 9 01:35:12.024962870 (376h24m46.795604205s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.024969803 (376h24m46.795611138s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.025076478 (376h24m46.795717813s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK): protocol: expected piece hash; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; EOF Jul 9 01:35:12.025102417 (376h24m46.795743742s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3): protocol: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection Jul 9 01:35:12.025112908 (376h24m46.795754233s) [seg_pos={0 0}, upload=301130]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:35:12.025165398 (376h24m46.795806734s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:35:12.025169095 (376h24m46.79581043s) [upload=301130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:35:12.344489789 (376h24m47.115131124s) [upload=301130]: Done uploading segments... Jul 9 01:35:12.344513775 (376h24m47.115155141s) [upload=301130]: Upload finished. err: <nil> Duration:* 40.085263996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.................................X......XX.XX..X......XXX...] duration: 47.9705643s FAIL (2025-07-09 01:38:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.970578928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X........................X...................] duration: 5.789287792s FAIL (2025-07-09 01:38:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ommit Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments... Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil> Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments... Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment... Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment. Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments... Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments... Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.789314454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................X..........................X......X...] duration: 29.065244043s FAIL (2025-07-09 01:37:40 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 9 01:34:13.079791339 (376h23m47.850432674s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:34:13.539243416 (376h23m48.309884751s) [seg_pos={0 0}, upload=301130]: Begin upload segment... Jul 9 01:34:13.539291719 (376h23m48.309933054s) [seg_pos={0 0}, upload=301130]: Done begin upload segment. Jul 9 01:34:13.539316947 (376h23m48.309958272s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539357194 (376h23m48.309998529s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539656567 (376h23m48.310297902s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539917576 (376h23m48.310558901s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539962171 (376h23m48.310603496s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540198754 (376h23m48.310840089s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540273528 (376h23m48.310914863s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540441268 (376h23m48.311082603s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540606304 (376h23m48.311247639s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540682078 (376h23m48.311323413s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540939712 (376h23m48.311581047s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540945533 (376h23m48.311586868s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541138272 (376h23m48.311779607s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541241369 (376h23m48.311882704s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541327143 (376h23m48.311968478s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541504031 (376h23m48.312145366s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541533798 (376h23m48.312175133s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541792674 (376h23m48.312433999s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:15.115895415 (376h23m49.88653675s) [upload=301130]: Next returned nil segment Jul 9 01:34:15.115912727 (376h23m49.886554062s) [upload=301130]: Waiting for error group managing segments... Jul 9 01:34:15.520996542 (376h23m50.291637867s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.527850836 (376h23m50.298492171s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529225055 (376h23m50.29986639s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529574193 (376h23m50.300215528s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.532332300 (376h23m50.302973635s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.534764755 (376h23m50.30540608s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.537145048 (376h23m50.307786383s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.542288250 (376h23m50.312929585s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.543383766 (376h23m50.314025101s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.558874324 (376h23m50.329515649s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.565242998 (376h23m50.335884333s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:25.557055824 (376h24m0.327697149s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:51.372897374 (376h24m26.143538709s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024816982 (376h24m46.795458317s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024848702 (376h24m46.795490037s) [seg_pos={0 0}, upload=301130]: Segment reached optimal threshold of 14 pieces. Jul 9 01:35:12.024962870 (376h24m46.795604205s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.024969803 (376h24m46.795611138s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.025076478 (376h24m46.795717813s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK): protocol: expected piece hash; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; EOF Jul 9 01:35:12.025102417 (376h24m46.795743742s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3): protocol: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection Jul 9 01:35:12.025112908 (376h24m46.795754233s) [seg_pos={0 0}, upload=301130]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:35:12.025165398 (376h24m46.795806734s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:35:12.025169095 (376h24m46.79581043s) [upload=301130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:35:12.344489789 (376h24m47.115131124s) [upload=301130]: Done uploading segments... Jul 9 01:35:12.344513775 (376h24m47.115155141s) [upload=301130]: Upload finished. err: <nil> Duration:* 29.065290573s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..................................X......XX.XX..X......XXX..] duration: 52.308101515s FAIL (2025-07-09 01:37:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.3081249s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X...X......XX....X...X.X....X...X.X..X...X..X...........] duration: 5.951462066s FAIL (2025-07-09 01:37:18 UTC) orbiter warning: context deadline exceeded Duration: 5.951482235s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................X..........................X......X..] duration: 54.373945793s FAIL (2025-07-09 01:36:58 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 9 01:34:13.079791339 (376h23m47.850432674s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:34:13.539243416 (376h23m48.309884751s) [seg_pos={0 0}, upload=301130]: Begin upload segment... Jul 9 01:34:13.539291719 (376h23m48.309933054s) [seg_pos={0 0}, upload=301130]: Done begin upload segment. Jul 9 01:34:13.539316947 (376h23m48.309958272s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539357194 (376h23m48.309998529s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539656567 (376h23m48.310297902s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539917576 (376h23m48.310558901s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539962171 (376h23m48.310603496s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540198754 (376h23m48.310840089s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540273528 (376h23m48.310914863s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540441268 (376h23m48.311082603s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540606304 (376h23m48.311247639s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540682078 (376h23m48.311323413s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540939712 (376h23m48.311581047s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540945533 (376h23m48.311586868s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541138272 (376h23m48.311779607s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541241369 (376h23m48.311882704s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541327143 (376h23m48.311968478s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541504031 (376h23m48.312145366s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541533798 (376h23m48.312175133s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541792674 (376h23m48.312433999s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:15.115895415 (376h23m49.88653675s) [upload=301130]: Next returned nil segment Jul 9 01:34:15.115912727 (376h23m49.886554062s) [upload=301130]: Waiting for error group managing segments... Jul 9 01:34:15.520996542 (376h23m50.291637867s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.527850836 (376h23m50.298492171s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529225055 (376h23m50.29986639s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529574193 (376h23m50.300215528s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.532332300 (376h23m50.302973635s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.534764755 (376h23m50.30540608s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.537145048 (376h23m50.307786383s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.542288250 (376h23m50.312929585s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.543383766 (376h23m50.314025101s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.558874324 (376h23m50.329515649s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.565242998 (376h23m50.335884333s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:25.557055824 (376h24m0.327697149s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:51.372897374 (376h24m26.143538709s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024816982 (376h24m46.795458317s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024848702 (376h24m46.795490037s) [seg_pos={0 0}, upload=301130]: Segment reached optimal threshold of 14 pieces. Jul 9 01:35:12.024962870 (376h24m46.795604205s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.024969803 (376h24m46.795611138s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.025076478 (376h24m46.795717813s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK): protocol: expected piece hash; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; EOF Jul 9 01:35:12.025102417 (376h24m46.795743742s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3): protocol: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection Jul 9 01:35:12.025112908 (376h24m46.795754233s) [seg_pos={0 0}, upload=301130]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:35:12.025165398 (376h24m46.795806734s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:35:12.025169095 (376h24m46.79581043s) [upload=301130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:35:12.344489789 (376h24m47.115131124s) [upload=301130]: Done uploading segments... Jul 9 01:35:12.344513775 (376h24m47.115155141s) [upload=301130]: Upload finished. err: <nil> Duration:* 54.373961413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X........................X..................] duration: 5.737506867s FAIL (2025-07-09 01:36:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ommit Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments... Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil> Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments... Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment... Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment. Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments... Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments... Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.737532836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X........................X.................] duration: 5.850711941s FAIL (2025-07-09 01:36:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ommit Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments... Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil> Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments... Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment... Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment. Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments... Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments... Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.850737299s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...................................X......XX.XX..X......XXX.] duration: 42.524344177s FAIL (2025-07-09 01:36:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.524360628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X...X......XX....X...X.X....X...X.X..X...X..X..........] duration: 5.998869352s FAIL (2025-07-09 01:36:06 UTC) orbiter warning: context deadline exceeded Duration: 5.998884039s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................X..........................X......X.] duration: 51.279194296s FAIL (2025-07-09 01:36:03 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 9 01:34:13.079791339 (376h23m47.850432674s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:34:13.539243416 (376h23m48.309884751s) [seg_pos={0 0}, upload=301130]: Begin upload segment... Jul 9 01:34:13.539291719 (376h23m48.309933054s) [seg_pos={0 0}, upload=301130]: Done begin upload segment. Jul 9 01:34:13.539316947 (376h23m48.309958272s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539357194 (376h23m48.309998529s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539656567 (376h23m48.310297902s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539917576 (376h23m48.310558901s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539962171 (376h23m48.310603496s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540198754 (376h23m48.310840089s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540273528 (376h23m48.310914863s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540441268 (376h23m48.311082603s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540606304 (376h23m48.311247639s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540682078 (376h23m48.311323413s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540939712 (376h23m48.311581047s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540945533 (376h23m48.311586868s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541138272 (376h23m48.311779607s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541241369 (376h23m48.311882704s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541327143 (376h23m48.311968478s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541504031 (376h23m48.312145366s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541533798 (376h23m48.312175133s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541792674 (376h23m48.312433999s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:15.115895415 (376h23m49.88653675s) [upload=301130]: Next returned nil segment Jul 9 01:34:15.115912727 (376h23m49.886554062s) [upload=301130]: Waiting for error group managing segments... Jul 9 01:34:15.520996542 (376h23m50.291637867s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.527850836 (376h23m50.298492171s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529225055 (376h23m50.29986639s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529574193 (376h23m50.300215528s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.532332300 (376h23m50.302973635s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.534764755 (376h23m50.30540608s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.537145048 (376h23m50.307786383s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.542288250 (376h23m50.312929585s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.543383766 (376h23m50.314025101s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.558874324 (376h23m50.329515649s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.565242998 (376h23m50.335884333s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:25.557055824 (376h24m0.327697149s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:51.372897374 (376h24m26.143538709s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024816982 (376h24m46.795458317s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024848702 (376h24m46.795490037s) [seg_pos={0 0}, upload=301130]: Segment reached optimal threshold of 14 pieces. Jul 9 01:35:12.024962870 (376h24m46.795604205s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.024969803 (376h24m46.795611138s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.025076478 (376h24m46.795717813s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK): protocol: expected piece hash; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; EOF Jul 9 01:35:12.025102417 (376h24m46.795743742s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3): protocol: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection Jul 9 01:35:12.025112908 (376h24m46.795754233s) [seg_pos={0 0}, upload=301130]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:35:12.025165398 (376h24m46.795806734s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:35:12.025169095 (376h24m46.79581043s) [upload=301130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:35:12.344489789 (376h24m47.115131124s) [upload=301130]: Done uploading segments... Jul 9 01:35:12.344513775 (376h24m47.115155141s) [upload=301130]: Upload finished. err: <nil> Duration:* 51.27921278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X........................X................] duration: 5.696399035s FAIL (2025-07-09 01:35:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ommit Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments... Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil> Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments... Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment... Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment. Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments... Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments... Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.696426786s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.....X...X......XX....X...X.X....X...X.X..X...X..X.........] duration: 18.417937723s FAIL (2025-07-09 01:35:38 UTC) orbiter warning: context deadline exceeded Duration: 18.417959124s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [....................................X......XX.XX..X......XXX] duration: 1m0.001446592s FAIL (2025-07-09 01:35:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001478503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................X..........................X......X] duration: 1m0.00235887s FAIL (2025-07-09 01:35:12 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 9 01:34:13.079791339 (376h23m47.850432674s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:34:13.539243416 (376h23m48.309884751s) [seg_pos={0 0}, upload=301130]: Begin upload segment... Jul 9 01:34:13.539291719 (376h23m48.309933054s) [seg_pos={0 0}, upload=301130]: Done begin upload segment. Jul 9 01:34:13.539316947 (376h23m48.309958272s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539357194 (376h23m48.309998529s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539656567 (376h23m48.310297902s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539917576 (376h23m48.310558901s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.539962171 (376h23m48.310603496s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540198754 (376h23m48.310840089s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540273528 (376h23m48.310914863s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540441268 (376h23m48.311082603s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540606304 (376h23m48.311247639s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540682078 (376h23m48.311323413s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540939712 (376h23m48.311581047s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.540945533 (376h23m48.311586868s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541138272 (376h23m48.311779607s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541241369 (376h23m48.311882704s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541327143 (376h23m48.311968478s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541504031 (376h23m48.312145366s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541533798 (376h23m48.312175133s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:13.541792674 (376h23m48.312433999s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Uploading piece... Jul 9 01:34:15.115895415 (376h23m49.88653675s) [upload=301130]: Next returned nil segment Jul 9 01:34:15.115912727 (376h23m49.886554062s) [upload=301130]: Waiting for error group managing segments... Jul 9 01:34:15.520996542 (376h23m50.291637867s) [noise=true , address=79.127.219.48:4076 , piece_id=OKS43AY72RTSEJ66KF2CJ5TW6ERBGOCO7M2PORQPU4QMJX6KGRHA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.527850836 (376h23m50.298492171s) [noise=true , address=77.90.31.12:4613 , piece_id=W235AQRRKF6VRV2DPLOTNCBP56XQ7X3KQKHWZMIS3AKA4BZ4KEXQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529225055 (376h23m50.29986639s) [noise=true , address=77.90.31.13:4640 , piece_id=6OWEPYEQ5CSHINBUXPNCE5REM62JE2YHT4OXC3NRE4AXV4MUMMCQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.529574193 (376h23m50.300215528s) [noise=true , address=77.90.31.18:4822 , piece_id=UXNH6XU2L5SEQKGSF22FDRHXLY2SVDNN4YHVGDPF3HMKOYQURSAQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.532332300 (376h23m50.302973635s) [noise=true , address=77.90.31.21:4883 , piece_id=IIMVVQI7O5B7QEABD77JEFY7GHZW7N4QXLCHQBRJCMHBV3XBBU4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.534764755 (376h23m50.30540608s) [noise=true , address=77.90.31.14:4688 , piece_id=BLHRV3UTJW4IJZ2MFFTDWKX2BP4Q3CRXOPGEGNJ4I2USW5X7LUNQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.537145048 (376h23m50.307786383s) [noise=true , address=77.90.31.19:4870 , piece_id=R6CDZ6S3KFAQ3II6ILNZWQBK2OLIQN3RWCRE2T7C4FTRNGUXR4VQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.542288250 (376h23m50.312929585s) [noise=true , address=77.90.31.15:4724 , piece_id=FWXVEGSBUBAPOZ5YGBDUR4LLP3HEIS35GKCKWNX56CCEIWIFDP2Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.543383766 (376h23m50.314025101s) [noise=true , address=77.90.31.19:4860 , piece_id=HTKSVWRQO7MZHYZHAHTN63NC7BGVERJIBXWTJDRHIA4QDIV7N3WA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.558874324 (376h23m50.329515649s) [noise=true , address=77.90.31.10:4533 , piece_id=BOI44N3KIOU7EIFEHNY32Z2CW4JWSFD6ICWPZTZE7RUX5G6IKBOA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:15.565242998 (376h23m50.335884333s) [noise=true , address=77.90.31.17:4791 , piece_id=67EZJK4MFFX6EM44MMDP64IIHQMRL77NJVOBQU52FEYB4BTQCYMQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:25.557055824 (376h24m0.327697149s) [noise=true , address=104.36.82.51:4051 , piece_id=6F2LU3WS5DE5KDN4EO4Y2IFSXHTSOMM6CLBDRCYTWVFOF43INGQA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:34:51.372897374 (376h24m26.143538709s) [noise=true , address=104.36.82.21:4488 , piece_id=2KXORGFFFJXEGVLIU6FOAQAE32MOS3VL3Q63WHCZJHZ4D5URYV5A, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024816982 (376h24m46.795458317s) [noise=true , address=104.36.82.8:4980 , piece_id=OUD375HCTFFU6EA7CGBOCOW67DPWS7337376KD3RHTBDNWJIFVEQ, seg_pos={0 0}, upload=301130]: Done uploading piece. err: <nil> Jul 9 01:35:12.024848702 (376h24m46.795490037s) [seg_pos={0 0}, upload=301130]: Segment reached optimal threshold of 14 pieces. Jul 9 01:35:12.024962870 (376h24m46.795604205s) [noise=true , address=104.36.82.50:4017 , piece_id=5EGKJ4EBVCB4PZVB2KIZTXBZTCWIYGMJVY2G4FHVV4ID7KDR76DA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.024969803 (376h24m46.795611138s) [noise=true , address=104.36.82.60:4372 , piece_id=4KL3P3D2RONFRAWYN7U2343DJMS5RVWEONU4U52SCIOKXEQKVANA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jul 9 01:35:12.025076478 (376h24m46.795717813s) [noise=true , address=104.36.82.19:4417 , piece_id=Z674RZ4K6ORMDVMFDRNQEF5VCS77OM5AEEX72SP2TXEG5XF55F4Q, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK): protocol: expected piece hash; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; write tcp 188.166.225.170:32916->104.36.82.19:4417: use of closed network connection; EOF Jul 9 01:35:12.025102417 (376h24m46.795743742s) [noise=true , address=104.36.82.40:4962 , piece_id=XREV2NVULU7PIUZRGAAC4VUUNZ3KOVKDSM5RXOVS7G2CLKXQYCPA, seg_pos={0 0}, upload=301130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3): protocol: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45180->104.36.82.40:4962: use of closed network connection Jul 9 01:35:12.025112908 (376h24m46.795754233s) [seg_pos={0 0}, upload=301130]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:35:12.025165398 (376h24m46.795806734s) [upload=301130]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:35:12.025169095 (376h24m46.79581043s) [upload=301130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:35:12.344489789 (376h24m47.115131124s) [upload=301130]: Done uploading segments... Jul 9 01:35:12.344513775 (376h24m47.115155141s) [upload=301130]: Upload finished. err: <nil> Duration:* 1m0.002370342s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.....X...X......XX....X...X.X....X...X.X..X...X..X........] duration: 10.41512868s FAIL (2025-07-09 01:35:03 UTC) orbiter warning: context deadline exceeded Duration: 10.415155201s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.....X...X......XX....X...X.X....X...X.X..X...X..X.......] duration: 23.884800581s FAIL (2025-07-09 01:34:40 UTC) orbiter warning: context deadline exceeded Duration: 23.884823885s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X........................X...............] duration: 5.803044348s FAIL (2025-07-09 01:34:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ommit Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments... Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil> Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments... Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment... Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment. Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments... Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments... Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.803069155s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....................................X......XX.XX..X......XX] duration: 1m0.001290162s FAIL (2025-07-09 01:34:32 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001302466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..........................X......] duration: 32.807275282s FAIL (2025-07-09 01:34:12 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== RequestItem_SegmentBegin Jul 9 01:29:01.337433896 (376h18m36.108075261s) [seg_pos={0 0}, upload=301063]: Begin upload segment... Jul 9 01:29:01.337487208 (376h18m36.108128543s) [seg_pos={0 0}, upload=301063]: Done begin upload segment. Jul 9 01:29:01.337511635 (376h18m36.10815298s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.337520201 (376h18m36.108161546s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338221482 (376h18m36.108862827s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338235780 (376h18m36.108877145s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338701732 (376h18m36.109343067s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338733382 (376h18m36.109374717s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339087590 (376h18m36.109728925s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339094813 (376h18m36.109736159s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339465213 (376h18m36.110106558s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339475853 (376h18m36.110117198s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339842684 (376h18m36.110484019s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339849308 (376h18m36.110490643s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.340181744 (376h18m36.110823079s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.340191722 (376h18m36.110833057s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.340777823 (376h18m36.111419158s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.341080963 (376h18m36.111722308s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.342101286 (376h18m36.112742631s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.344027331 (376h18m36.114668667s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:02.921702307 (376h18m37.692343642s) [upload=301063]: Next returned nil segment Jul 9 01:29:02.921717205 (376h18m37.69235854s) [upload=301063]: Waiting for error group managing segments... Jul 9 01:29:03.330770575 (376h18m38.10141191s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.333021060 (376h18m38.103662395s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.346762402 (376h18m38.117403737s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.346815323 (376h18m38.117456658s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347060412 (376h18m38.117701747s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347185391 (376h18m38.117826726s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347425851 (376h18m38.118067186s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347529449 (376h18m38.118170784s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347923393 (376h18m38.118564728s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.348515206 (376h18m38.119156541s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.363502611 (376h18m38.134143946s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:26.444658884 (376h19m1.215300209s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:37.476126743 (376h19m12.246768078s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:56.449247029 (376h19m31.219888364s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:56.449272748 (376h19m31.219914073s) [seg_pos={0 0}, upload=301063]: Segment reached optimal threshold of 14 pieces. Jul 9 01:29:56.449354975 (376h19m31.2199963s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Jul 9 01:29:56.449457422 (376h19m31.220098757s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF Jul 9 01:29:56.449486968 (376h19m31.220128303s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MHmmjheT4VDX93MiQNzQyTWkxZFzz2PwNvdRfsevZRbj6qmJS): protocol: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection Jul 9 01:29:56.449513970 (376h19m31.220155295s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Pua5x8rAsuqYWCgoAY6fgy2SXNoDjzt7WwgMhzv1HN2KG5NY4): protocol: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection Jul 9 01:29:56.449528637 (376h19m31.220169972s) [seg_pos={0 0}, upload=301063]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:29:56.449590266 (376h19m31.220231601s) [upload=301063]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:29:56.449593853 (376h19m31.220235178s) [upload=301063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:29:56.719973407 (376h19m31.490614742s) [upload=301063]: Done uploading segments... Jul 9 01:29:56.719991001 (376h19m31.490632336s) [upload=301063]: Upload finished. err: <nil> Duration: 32.80730044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X........................X..............] duration: 5.822145727s FAIL (2025-07-09 01:34:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ommit Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments... Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil> Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments... Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment... Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment. Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments... Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments... Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.822166065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X........................X.............] duration: 24.719262643s FAIL (2025-07-09 01:33:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ommit Jul 9 01:22:25.859678203 (376h12m0.630319538s) [upload=300976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:22:26.159265421 (376h12m0.929906756s) [upload=300976]: Done uploading segments... Jul 9 01:22:26.159284298 (376h12m0.929925623s) [upload=300976]: Upload finished. err: <nil> Jul 9 01:23:00.953905336 (376h12m35.724546671s) [upload=300988]: Starting upload Jul 9 01:23:00.953962656 (376h12m35.724603991s) [upload=300988]: Done starting upload Jul 9 01:23:00.954088757 (376h12m35.724730092s) [upload=300988]: Uploading segments... Jul 9 01:23:00.954179911 (376h12m35.724821256s) [upload=300988]: Got next segment. Inline: false Jul 9 01:23:00.954239456 (376h12m35.724880801s) [upload=300988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:23:00.954244716 (376h12m35.724886051s) [upload=300988]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:23:01.237284104 (376h12m36.00792544s) [seg_pos={0 0}, upload=300988]: Begin upload segment... Jul 9 01:23:01.237331095 (376h12m36.00797243s) [seg_pos={0 0}, upload=300988]: Done begin upload segment. Jul 9 01:23:01.237361082 (376h12m36.008002417s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237365891 (376h12m36.008007226s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237776807 (376h12m36.008418142s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.237790815 (376h12m36.00843215s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238358791 (376h12m36.009000126s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238724561 (376h12m36.009365896s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.238859569 (376h12m36.009500904s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239102363 (376h12m36.009743698s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239340349 (376h12m36.009981684s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239397749 (376h12m36.010039084s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.239650191 (376h12m36.010291527s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240053333 (376h12m36.010694668s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240216586 (376h12m36.010857921s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240437839 (376h12m36.011079184s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240628955 (376h12m36.01127029s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:01.240761428 (376h12m36.011402763s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Uploading piece... Jul 9 01:23:02.934794737 (376h12m37.705436072s) [upload=300988]: Next returned nil segment Jul 9 01:23:02.934829535 (376h12m37.70547087s) [upload=300988]: Waiting for error group managing segments... Jul 9 01:23:03.202814027 (376h12m37.973455362s) [noise=true , address=104.36.82.9:4006 , piece_id=OIQT3X5P5E3EUACADS6C2YDYBCFZUED6QYIIVLO4DS4FKJOHQF2A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.222356328 (376h12m37.992997663s) [noise=true , address=104.36.82.16:4295 , piece_id=NX6ITBLJI5PJVH5L7WTGLB6HVKD6V7AH4MEMUSMUMHZO5IFD5VWA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.238172479 (376h12m38.008813814s) [noise=true , address=104.36.82.16:4294 , piece_id=WVQGT5QLDAGYIVQ3B5UFOWPL4NMKEUWQV6JFQEU4WKI3R27EKX4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.314238495 (376h12m38.08487983s) [noise=true , address=45.91.116.4:4938 , piece_id=4EUTMOKLJ56EGBCS5NOSHIRQPB7AHCALT4WM33Y7JB6PQGZQNBCA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.368720175 (376h12m38.13936151s) [noise=true , address=77.90.31.17:4772 , piece_id=5ULTL7AX37OO6GWXL2W6T43RQITZJDRVDPQFH7NALU4N3Z2HJD4Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.383073236 (376h12m38.153714571s) [noise=true , address=77.90.31.13:4626 , piece_id=C6GMGD5APPQND472KVNMWZL6USXGOBREZ4KP4F6YNFJ47E5LCAZA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.388258438 (376h12m38.158899783s) [noise=true , address=77.90.31.16:4734 , piece_id=H65VNG4ITT5XJHFELJLU6QA64EFAM5PN6ME3HAJSDSRVYC5CS7OQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.397556317 (376h12m38.168197672s) [noise=true , address=45.91.116.10:4082 , piece_id=BWVDKKMS23QJLOWVO5EWC3O6Y7D3TFSNCZ3AFMDIHW3DGXU3B7RA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.406808579 (376h12m38.177449914s) [noise=true , address=45.91.116.6:4974 , piece_id=TWDE5I4CXR5YDOQQA4FJKNSUE3B4MW6I7IICHAX23IH2UWN35VVQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408467713 (376h12m38.179109048s) [noise=true , address=45.91.116.14:4189 , piece_id=OD6T4RMNHHMTMB25RMNXDYU74AMHAYQE7RJMFAWL5KNA2WUWG3WQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:03.408511106 (376h12m38.179152442s) [noise=true , address=45.91.116.5:4299 , piece_id=7MFTAXCED2A5NWMCE3I23O7KGLOGMIXSPISHMIV76B36LQ5Q4UYQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:25.064475081 (376h12m59.835116416s) [noise=true , address=104.36.82.13:4186 , piece_id=TWUBNHYRAOCPKG3FWG2OP5REF7SED73O7DGP3XYAEWIPPSW6MK7Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: <nil> Jul 9 01:23:30.310395800 (376h13m5.081037135s) [noise=true , address=104.36.82.12:4149 , piece_id=MTFWOMU7HD7AFI5OEVQUMIFSVBTPPLNZQFLCAG5ZSMRWAJCZ5RQQ, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->104.36.82.12:4149: use of closed network connection Jul 9 01:23:30.310447889 (376h13m5.081089225s) [noise=true , address=104.36.82.5:4838 , piece_id=7WTUKTX3OJ33KHKTQYWXEHY5JC33OC36WLYUVMXLER7Q3OBGEJ5A, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:58624->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 01:23:30.310485371 (376h13m5.081126706s) [noise=true , address=104.36.82.7:4909 , piece_id=KSOVFVAKXY5DZWR3JGSWYQ4OEXSJLOO643XK5BP3IYABJMYPA32Q, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41904->104.36.82.7:4909: use of closed network connection Jul 9 01:23:30.310515829 (376h13m5.081157164s) [noise=true , address=104.36.82.10:4078 , piece_id=5J4GZHYKP6RFSBN4ZOZI3RHPJSVTGSNS64IB7TFN6NIPAERS2MHA, seg_pos={0 0}, upload=300988]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57122->104.36.82.10:4078: use of closed network connection Jul 9 01:23:30.310530367 (376h13m5.081171702s) [seg_pos={0 0}, upload=300988]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:23:30.543933606 (376h13m5.314574991s) [upload=300988]: Done uploading segments... Jul 9 01:23:30.543963884 (376h13m5.314605259s) [upload=300988]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 24.719296958s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X.....X...X......XX....X...X.X....X...X.X..X...X..X......] duration: 16.075579877s FAIL (2025-07-09 01:33:47 UTC) orbiter warning: context deadline exceeded Duration: 16.075603712s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..........................X.....] duration: 28.585352446s FAIL (2025-07-09 01:33:39 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== RequestItem_SegmentBegin Jul 9 01:29:01.337433896 (376h18m36.108075261s) [seg_pos={0 0}, upload=301063]: Begin upload segment... Jul 9 01:29:01.337487208 (376h18m36.108128543s) [seg_pos={0 0}, upload=301063]: Done begin upload segment. Jul 9 01:29:01.337511635 (376h18m36.10815298s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.337520201 (376h18m36.108161546s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338221482 (376h18m36.108862827s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338235780 (376h18m36.108877145s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338701732 (376h18m36.109343067s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338733382 (376h18m36.109374717s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339087590 (376h18m36.109728925s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339094813 (376h18m36.109736159s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339465213 (376h18m36.110106558s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339475853 (376h18m36.110117198s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339842684 (376h18m36.110484019s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339849308 (376h18m36.110490643s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.340181744 (376h18m36.110823079s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.340191722 (376h18m36.110833057s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.340777823 (376h18m36.111419158s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.341080963 (376h18m36.111722308s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.342101286 (376h18m36.112742631s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.344027331 (376h18m36.114668667s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:02.921702307 (376h18m37.692343642s) [upload=301063]: Next returned nil segment Jul 9 01:29:02.921717205 (376h18m37.69235854s) [upload=301063]: Waiting for error group managing segments... Jul 9 01:29:03.330770575 (376h18m38.10141191s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.333021060 (376h18m38.103662395s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.346762402 (376h18m38.117403737s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.346815323 (376h18m38.117456658s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347060412 (376h18m38.117701747s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347185391 (376h18m38.117826726s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347425851 (376h18m38.118067186s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347529449 (376h18m38.118170784s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347923393 (376h18m38.118564728s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.348515206 (376h18m38.119156541s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.363502611 (376h18m38.134143946s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:26.444658884 (376h19m1.215300209s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:37.476126743 (376h19m12.246768078s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:56.449247029 (376h19m31.219888364s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:56.449272748 (376h19m31.219914073s) [seg_pos={0 0}, upload=301063]: Segment reached optimal threshold of 14 pieces. Jul 9 01:29:56.449354975 (376h19m31.2199963s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Jul 9 01:29:56.449457422 (376h19m31.220098757s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF Jul 9 01:29:56.449486968 (376h19m31.220128303s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MHmmjheT4VDX93MiQNzQyTWkxZFzz2PwNvdRfsevZRbj6qmJS): protocol: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection Jul 9 01:29:56.449513970 (376h19m31.220155295s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Pua5x8rAsuqYWCgoAY6fgy2SXNoDjzt7WwgMhzv1HN2KG5NY4): protocol: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection Jul 9 01:29:56.449528637 (376h19m31.220169972s) [seg_pos={0 0}, upload=301063]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:29:56.449590266 (376h19m31.220231601s) [upload=301063]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:29:56.449593853 (376h19m31.220235178s) [upload=301063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:29:56.719973407 (376h19m31.490614742s) [upload=301063]: Done uploading segments... Jul 9 01:29:56.719991001 (376h19m31.490632336s) [upload=301063]: Upload finished. err: <nil> Duration: 28.585369008s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [......................................X......XX.XX..X......X] duration: 1m0.000858884s FAIL (2025-07-09 01:33:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000879382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X..........................X....] duration: 45.154087067s FAIL (2025-07-09 01:33:10 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== RequestItem_SegmentBegin Jul 9 01:29:01.337433896 (376h18m36.108075261s) [seg_pos={0 0}, upload=301063]: Begin upload segment... Jul 9 01:29:01.337487208 (376h18m36.108128543s) [seg_pos={0 0}, upload=301063]: Done begin upload segment. Jul 9 01:29:01.337511635 (376h18m36.10815298s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.337520201 (376h18m36.108161546s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338221482 (376h18m36.108862827s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338235780 (376h18m36.108877145s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338701732 (376h18m36.109343067s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.338733382 (376h18m36.109374717s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339087590 (376h18m36.109728925s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339094813 (376h18m36.109736159s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339465213 (376h18m36.110106558s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339475853 (376h18m36.110117198s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339842684 (376h18m36.110484019s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.339849308 (376h18m36.110490643s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.340181744 (376h18m36.110823079s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.340191722 (376h18m36.110833057s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.340777823 (376h18m36.111419158s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.341080963 (376h18m36.111722308s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.342101286 (376h18m36.112742631s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:01.344027331 (376h18m36.114668667s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Uploading piece... Jul 9 01:29:02.921702307 (376h18m37.692343642s) [upload=301063]: Next returned nil segment Jul 9 01:29:02.921717205 (376h18m37.69235854s) [upload=301063]: Waiting for error group managing segments... Jul 9 01:29:03.330770575 (376h18m38.10141191s) [noise=true , address=77.90.31.11:4581 , piece_id=TKMT65PF3VOTCHFY6GBFC2W5JYOJTAHK5WFRZF3YAPOYHA7SGFDQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.333021060 (376h18m38.103662395s) [noise=true , address=77.90.31.15:4701 , piece_id=M6US4USD57GPEL35W4IJF73X3BJMDRBLD2UXSHTLTYHMVI6XCTTA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.346762402 (376h18m38.117403737s) [noise=true , address=77.90.31.14:4682 , piece_id=KWYECXGPZ4TVAXVUK5IJHYRW36OFZGXFRWDLUSPJNZUUO5JDGLZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.346815323 (376h18m38.117456658s) [noise=true , address=77.90.31.11:4563 , piece_id=BC47C6JJI2QGZT54RFSVTB34IWFZVY2MYP4R55HYM3MC6PZ63L3Q, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347060412 (376h18m38.117701747s) [noise=true , address=77.90.31.17:4786 , piece_id=X4VZDLZC2AAXL5VRABJE5PIU6RJGJOQH67VHMSLT4VPLMF64QQ2A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347185391 (376h18m38.117826726s) [noise=true , address=77.90.31.21:4909 , piece_id=BDRGEIFZ56SQ46453YLMKGZZE5F4FHIUJE35RU5RCUNDNQFXYSPQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347425851 (376h18m38.118067186s) [noise=true , address=77.90.31.13:4651 , piece_id=TF3NGJ4AASQSXY4JSF27FCW7VSHF5VWYLHMNXHEQAMT5KJWXVNQQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347529449 (376h18m38.118170784s) [noise=true , address=77.90.31.12:4606 , piece_id=JCQCFDBA4YSIFSKS4RFYFUUYUAFU3XR2KI7VV4FCGFD4POFGXCHA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.347923393 (376h18m38.118564728s) [noise=true , address=77.90.31.10:4537 , piece_id=WQKTKYRY4C32EGPMHSXQAS3KSRAWIDDF7BYBZ2MKZOWHKFEXBGLA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.348515206 (376h18m38.119156541s) [noise=true , address=77.90.31.16:4757 , piece_id=HUJGTDFFEA5ATNB2HXCBERZYDL5N5BLMZYW6G3DASQTC2PM4IFYA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:03.363502611 (376h18m38.134143946s) [noise=true , address=77.90.31.10:4533 , piece_id=CMUMWPDGLP2XWKFHCSXNE3IB7HYJEDOVYQNUM7DKI7THVBIHNDHQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:26.444658884 (376h19m1.215300209s) [noise=true , address=104.36.82.19:4433 , piece_id=VUXOICADHIDKYIDNKJXDN3WA6FNDG3XJR3OCXC4UC26EA5P3NIMA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:37.476126743 (376h19m12.246768078s) [noise=true , address=104.36.82.36:4651 , piece_id=QIMZ4CWZJCOYIIWEZGQYWZ2EHHL7DYAHHQ2YX5W7PPCH4QJNWS4A, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:56.449247029 (376h19m31.219888364s) [noise=true , address=104.36.82.5:4868 , piece_id=W7Q3QS3IKB7X6TOOSKKMX5W42BHG274PXYNZ7UZCGSPNQXUFAURQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: <nil> Jul 9 01:29:56.449272748 (376h19m31.219914073s) [seg_pos={0 0}, upload=301063]: Segment reached optimal threshold of 14 pieces. Jul 9 01:29:56.449354975 (376h19m31.2199963s) [noise=true , address=104.36.82.21:4477 , piece_id=N4BEJUNAM2RKLEZ7A7XKJI525EBWQ35UBMMSBZLIIC5AXNLMXCKA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Jul 9 01:29:56.449457422 (376h19m31.220098757s) [noise=true , address=104.36.82.44:4880 , piece_id=44M5JZ2MTKYBK6TH5HX46KAL24UCXIO4X4PV6DY6VUE4JHGKJXRA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF Jul 9 01:29:56.449486968 (376h19m31.220128303s) [noise=true , address=104.36.82.18:4391 , piece_id=FI7VKWZELHQFI2G4R5GOPAM4NLRT4VKGBJ26POQ2IS24F5EW77ZA, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MHmmjheT4VDX93MiQNzQyTWkxZFzz2PwNvdRfsevZRbj6qmJS): protocol: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57052->104.36.82.18:4391: use of closed network connection Jul 9 01:29:56.449513970 (376h19m31.220155295s) [noise=true , address=104.36.82.35:4614 , piece_id=JTXUCFS36ZNLCZHAEZORW77F3ZNAN5M6FYKQ2ZFYUAFXY5DRC3GQ, seg_pos={0 0}, upload=301063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Pua5x8rAsuqYWCgoAY6fgy2SXNoDjzt7WwgMhzv1HN2KG5NY4): protocol: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49282->104.36.82.35:4614: use of closed network connection Jul 9 01:29:56.449528637 (376h19m31.220169972s) [seg_pos={0 0}, upload=301063]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 01:29:56.449590266 (376h19m31.220231601s) [upload=301063]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:29:56.449593853 (376h19m31.220235178s) [upload=301063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:29:56.719973407 (376h19m31.490614742s) [upload=301063]: Done uploading segments... Jul 9 01:29:56.719991001 (376h19m31.490632336s) [upload=301063]: Upload finished. err: <nil> Duration: 45.154105223s core uplink us-select-1 us1-placement-20

1
...
8285
8286
8287
8288
8289
...
8565