Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.265048878s FAIL (2025-07-16 19:34:49 UTC) orbiter warning: context deadline exceeded Duration: 6.265067062s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration: 5.93055172s FAIL (2025-07-16 19:34:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.930579413s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.116277037s FAIL (2025-07-16 19:33:37 UTC) orbiter warning: context deadline exceeded Duration: 6.11629988s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 5.996792491s FAIL (2025-07-16 19:33:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.996819171s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.304465205s FAIL (2025-07-16 19:33:01 UTC) orbiter warning: context deadline exceeded Duration: 6.304480505s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 5.886203468s FAIL (2025-07-16 19:32:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.886223988s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.058531989s FAIL (2025-07-16 19:31:46 UTC) orbiter warning: context deadline exceeded Duration: 6.058561917s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.542301665s FAIL (2025-07-16 19:31:31 UTC) orbiter warning: context deadline exceeded Duration: 6.542320431s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.X........................................] duration: 5.947793098s FAIL (2025-07-16 19:31:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.9478199s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 5.821751628s FAIL (2025-07-16 19:31:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.821775362s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.229759486s FAIL (2025-07-16 19:30:37 UTC) orbiter warning: context deadline exceeded Duration: 6.229781239s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 6.022316832s FAIL (2025-07-16 19:30:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.022345957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 5.828156364s FAIL (2025-07-16 19:29:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.828179338s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.117877347s FAIL (2025-07-16 19:29:31 UTC) orbiter warning: context deadline exceeded Duration: 6.117899849s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.286233581s FAIL (2025-07-16 19:28:41 UTC) orbiter warning: context deadline exceeded Duration: 6.286250084s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 5.982947821s FAIL (2025-07-16 19:28:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.982983068s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.161780048s FAIL (2025-07-16 19:28:00 UTC) orbiter warning: context deadline exceeded Duration: 6.161804805s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 5.741180499s FAIL (2025-07-16 19:27:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.741211349s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.49207984s FAIL (2025-07-16 19:27:29 UTC) orbiter warning: context deadline exceeded Duration: 6.49210553s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.X..................................] duration: 5.797438679s FAIL (2025-07-16 19:27:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.797464488s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.044380806s FAIL (2025-07-16 19:26:53 UTC) orbiter warning: context deadline exceeded Duration: 6.04439856s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.X.................................] duration: 6.204922916s FAIL (2025-07-16 19:26:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.204966127s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.232572732s FAIL (2025-07-16 19:26:17 UTC) orbiter warning: context deadline exceeded Duration: 6.232594232s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.X................................] duration: 5.670318133s FAIL (2025-07-16 19:25:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.670341368s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.444387151s FAIL (2025-07-16 19:25:16 UTC) orbiter warning: context deadline exceeded Duration: 6.444409234s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.X...............................] duration: 5.859634344s FAIL (2025-07-16 19:24:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.859654692s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.39021907s FAIL (2025-07-16 19:24:45 UTC) orbiter warning: context deadline exceeded Duration: 6.390232857s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration: 5.87726397s FAIL (2025-07-16 19:24:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.877285531s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.307714261s FAIL (2025-07-16 19:23:45 UTC) orbiter warning: context deadline exceeded Duration: 6.307727486s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.X.............................] duration: 5.862549518s FAIL (2025-07-16 19:23:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.86257148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.X............................] duration: 5.718246019s FAIL (2025-07-16 19:23:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.718271648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 5.727382142s FAIL (2025-07-16 19:22:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.72740824s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.021282831s FAIL (2025-07-16 19:22:32 UTC) orbiter warning: context deadline exceeded Duration: 6.021332197s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..............................X.X..........................] duration: 5.833400868s FAIL (2025-07-16 19:21:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.833426297s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.256267867s FAIL (2025-07-16 19:21:43 UTC) orbiter warning: context deadline exceeded Duration: 6.25629004s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.088590946s FAIL (2025-07-16 19:21:19 UTC) orbiter warning: context deadline exceeded Duration: 6.088606224s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.010465219s FAIL (2025-07-16 19:20:49 UTC) orbiter warning: context deadline exceeded Duration: 6.010489778s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..............................X.X.........................] duration: 5.774578558s FAIL (2025-07-16 19:20:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.774615248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..............................X.X........................] duration: 5.810401263s FAIL (2025-07-16 19:20:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.810427635s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.280785275s FAIL (2025-07-16 19:19:44 UTC) orbiter warning: context deadline exceeded Duration: 6.280806956s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..............................X.X.......................] duration: 5.885035409s FAIL (2025-07-16 19:19:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.88506773s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.106593977s FAIL (2025-07-16 19:19:18 UTC) orbiter warning: context deadline exceeded Duration: 6.10661629s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.326767827s FAIL (2025-07-16 19:18:39 UTC) orbiter warning: context deadline exceeded Duration: 6.326793226s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..............................X.X......................] duration: 5.823824295s FAIL (2025-07-16 19:18:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.823855203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..............................X.X.....................] duration: 5.824609104s FAIL (2025-07-16 19:18:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.824640883s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.002989727s FAIL (2025-07-16 19:17:48 UTC) orbiter warning: context deadline exceeded Duration: 6.003006789s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..............................X.X....................] duration: 6.0196281s FAIL (2025-07-16 19:17:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.019649179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..............................X.X...................] duration: 6.44314145s FAIL (2025-07-16 19:16:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.443169634s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.970285652s FAIL (2025-07-16 19:16:35 UTC) orbiter warning: context deadline exceeded Duration: 5.970308516s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..............................X.X..................] duration: 5.790090986s FAIL (2025-07-16 19:16:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 16564 (561h53m24.798457899s) [upload=449313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 19:03:50.027820431 (561h53m24.798461756s) [upload=449313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 19:03:50.325177169 (561h53m25.095818494s) [upload=449313]: Done uploading segments... Jul 16 19:03:50.325193620 (561h53m25.095834955s) [upload=449313]: Upload finished. err: <nil> Jul 16 19:04:48.569836177 (561h54m23.340477512s) [upload=449326]: Starting upload Jul 16 19:04:48.569938393 (561h54m23.340579728s) [upload=449326]: Done starting upload Jul 16 19:04:48.570035328 (561h54m23.340676663s) [upload=449326]: Uploading segments... Jul 16 19:04:48.570178030 (561h54m23.340819366s) [upload=449326]: Got next segment. Inline: false Jul 16 19:04:48.570225461 (561h54m23.340866797s) [upload=449326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 19:04:48.570660875 (561h54m23.34130221s) [upload=449326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 19:04:48.802766945 (561h54m23.57340827s) [seg_pos={0 0}, upload=449326]: Begin upload segment... Jul 16 19:04:48.802809045 (561h54m23.57345038s) [seg_pos={0 0}, upload=449326]: Done begin upload segment. Jul 16 19:04:48.802829214 (561h54m23.573470549s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.802862217 (561h54m23.573503552s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803207267 (561h54m23.573848602s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803427409 (561h54m23.574068744s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803527329 (561h54m23.574168664s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803732292 (561h54m23.574373627s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.803830118 (561h54m23.574471453s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804030362 (561h54m23.574671697s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804134861 (561h54m23.574776196s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804411341 (561h54m23.575052676s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804626612 (561h54m23.575267947s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.804816937 (561h54m23.575458272s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805045004 (561h54m23.575686339s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805130928 (561h54m23.575772263s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805372630 (561h54m23.576013965s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:48.805623861 (561h54m23.576265196s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Uploading piece... Jul 16 19:04:50.496612514 (561h54m25.267253849s) [upload=449326]: Next returned nil segment Jul 16 19:04:50.496630077 (561h54m25.267271413s) [upload=449326]: Waiting for error group managing segments... Jul 16 19:04:50.859098237 (561h54m25.629739572s) [noise=true , address=104.36.82.8:4972 , piece_id=L7CXAJ2DUQIWVUIHBPLUVL43BMAHJWRPX732DPO6NRVKUDJCKJDQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.863702978 (561h54m25.634344313s) [noise=true , address=45.91.116.7:4009 , piece_id=TVUNFPNS3VOUFQ32YTH2RBRL66SGMDXFEU5BGBDHAPZEMN5M5IMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864453323 (561h54m25.635094658s) [noise=true , address=45.91.116.13:4154 , piece_id=UIJ6G5XGBVWYUSQEV65UDX27GPF4NDB52J7WZQNPSQ2LRRNJOWCA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864756403 (561h54m25.635397738s) [noise=true , address=104.36.82.8:4970 , piece_id=6U62DE2ZJQ27HK7QXRHWDQAYQCSE3JHGTDM3SHX3JLFVB57FUXOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.864829954 (561h54m25.635471289s) [noise=true , address=104.36.82.6:4876 , piece_id=F4D2YJLR2677R5WN4W27AO7PVY3DMDV7PK7S4YGHUZ3FYUFTCUPQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.929638304 (561h54m25.700279639s) [noise=true , address=104.36.82.9:4007 , piece_id=COPN7CHQV4HZP74KELOGKBSMUIBKUIV74TI4I3I5H2R4UZ4E2QMQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.933494202 (561h54m25.704135547s) [noise=true , address=104.36.82.13:4185 , piece_id=VBKPYYDBOJ6YYJCH2F4VRO7ZI4M2FN2NQ2H74ZPFG5WWYNW2NUBQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:50.991985657 (561h54m25.762626992s) [noise=true , address=104.36.82.16:4295 , piece_id=AC3PK7BZKTL2NACWPYFIBWQ2TL3BYU7BKRJRMGVBNQD6HOBAF2JQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.007934743 (561h54m25.778576078s) [noise=true , address=104.36.82.14:4222 , piece_id=GPJV36JGX6DTYLF5ITKSB7BCBX4KUANV4RKV5LRIJKLZ33XBRIWQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.010915507 (561h54m25.781556832s) [noise=true , address=104.36.82.11:4113 , piece_id=O7S74GXF5AX6L5XLNSQLNKY2HMDEM452TCGTAHPJLDQRXOR5X7UQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:04:51.011671332 (561h54m25.782312658s) [noise=true , address=104.36.82.4:4801 , piece_id=HKYKV4OF5C6PLFHV3ZRPKDE5S3OYRRAAKK2UPPEVRINLSJDFIM6A, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.583107142 (561h54m52.353748477s) [noise=true , address=77.90.31.19:4841 , piece_id=32BPDUKYQRIIPMSNAG6KNM2QG5AFMCFQD5RZZWFIUMCCU52BR4QQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: <nil> Jul 16 19:05:17.926380154 (561h54m52.697021489s) [noise=true , address=77.90.31.15:4697 , piece_id=4S6F332OXLEBTNRTKKSLG2CFI47H4WQKTFX35OL7HYIVDVYOVXBA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 19:05:17.926475696 (561h54m52.697117031s) [noise=true , address=77.90.31.19:4842 , piece_id=XPZNUBJORWYFO4CUYKVZR2CYDVOEVEFVPFUNGR5S554Y6VKYJ3OA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47984->77.90.31.19:4842: use of closed network connection Jul 16 19:05:17.926548556 (561h54m52.697189891s) [noise=true , address=77.90.31.18:4807 , piece_id=RWQPBP527NLG7LQKV2Q5KGCZHYPUOY64NCMWD3LZTKR4H5IB6HKA, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45628->77.90.31.18:4807: use of closed network connection Jul 16 19:05:17.926617057 (561h54m52.697258392s) [noise=true , address=77.90.31.12:4589 , piece_id=BPOO2PW5QE4O4PK25YONYV5FG257T6JNZH6BRD2GKAFLQVVNUMOQ, seg_pos={0 0}, upload=449326]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40108->77.90.31.12:4589: use of closed network connection Jul 16 19:05:17.926638227 (561h54m52.697279562s) [seg_pos={0 0}, upload=449326]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 19:05:18.156778165 (561h54m52.9274195s) [upload=449326]: Done uploading segments... Jul 16 19:05:18.156800027 (561h54m52.927441362s) [upload=449326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.790133408s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)