Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X..XX....X...X.X..X..XX...................................] duration: 11.17132289s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:07:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 11.171348128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X..X..XX....X...X.X..X..XX..................................] duration: 5.676472996s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:07:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.676508253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X..X..XX....X...X.X..X..XX.................................] duration: 5.945304666s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:06:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.945334372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X..X..XX....X...X.X..X..XX................................] duration: 6.467963033s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:06:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 6.467991537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X..X..XX....X...X.X..X..XX...............................] duration: 6.114117588s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:05:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 6.114179256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X..X..XX....X...X.X..X..XX..............................] duration: 5.997719938s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:04:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.997751969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X..X..XX....X...X.X..X..XX.............................] duration: 5.954039843s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:03:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.954061954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X..X..XX....X...X.X..X..XX............................] duration: 5.872486482s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:02:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.872511319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X..X..XX....X...X.X..X..XX...........................] duration: 6.236284115s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:01:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 6.236314493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........X..X..XX....X...X.X..X..XX..........................] duration: 9.802709743s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:00:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 9.802729111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........X..X..XX....X...X.X..X..XX.........................] duration: 5.882925589s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:59:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.882956658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........X..X..XX....X...X.X..X..XX........................] duration: 5.686182616s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:58:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.686209628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........X..X..XX....X...X.X..X..XX.......................] duration: 5.898644401s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:57:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.89867053s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X...........X..X..XX....X...X.X..X..XX......................] duration: 5.891637328s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:57:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.891663527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X...........X..X..XX....X...X.X..X..XX.....................] duration: 23.671481761s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:56:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 23.671515975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X...........X..X..XX....X...X.X..X..XX....................] duration: 5.937483176s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:55:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.937513673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X...........X..X..XX....X...X.X..X..XX...................] duration: 5.74768531s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:54:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.74776418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X...........X..X..XX....X...X.X..X..XX..................] duration: 5.870725849s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:54:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.870754373s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X...........X..X..XX....X...X.X..X..XX.................] duration: 5.992264261s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:53:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.992290751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X...........X..X..XX....X...X.X..X..XX................] duration: 8.915254603s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:52:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 8.915283107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......X...........X..X..XX....X...X.X..X..XX...............] duration: 5.912611327s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:51:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.912650131s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........X...........X..X..XX....X...X.X..X..XX..............] duration: 5.912631495s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:50:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.912661351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........X...........X..X..XX....X...X.X..X..XX.............] duration: 7.411469965s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:49:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 7.411499541s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.........X...........X..X..XX....X...X.X..X..XX............] duration: 7.127285939s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:49:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 7.127312358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.........X...........X..X..XX....X...X.X..X..XX...........] duration: 20.409527663s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:48:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 20.4095528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X.........X...........X..X..XX....X...X.X..X..XX..........] duration: 5.964754081s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:48:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.96478s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.........X...........X..X..XX....X...X.X..X..XX.........] duration: 6.159720429s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:47:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 6.159751057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.........X...........X..X..XX....X...X.X..X..XX........] duration: 6.354052811s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:45:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 6.354073181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.........X...........X..X..XX....X...X.X..X..XX.......] duration: 6.197624455s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:45:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 6.197658941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.....X.........X...........X..X..XX....X...X.X..X..XX......] duration: 5.690474806s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:44:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.690500776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.....X.........X...........X..X..XX....X...X.X..X..XX.....] duration: 26.970530945s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:43:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 26.970572102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.....X.........X...........X..X..XX....X...X.X..X..XX....] duration: 10.237241249s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:42:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 10.237269683s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.....X.........X...........X..X..XX....X...X.X..X..XX...] duration: 25.552792459s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:41:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 25.55281831s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...X.....X.........X...........X..X..XX....X...X.X..X..XX..] duration: 27.698538055s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:41:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 27.698569365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...X.....X.........X...........X..X..XX....X...X.X..X..XX.] duration: 8.903143368s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:39:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 8.90316587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X...X.....X.........X...........X..X..XX....X...X.X..X..XX] duration: 30.004913073s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 17:39:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 30.004941347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X...X.....X.........X...........X..X..XX....X...X.X..X..X] duration: 30.002196936s trace: https://jaeger.dev.storj.io/trace/d8214e38030cd6e FAIL (2025-12-22 17:38:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SIYZEDKL55H42OMOIP45LKLZAAGO7VC73VLM7TMMMCHI37CF3CBQ) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 4NQLBN4CGZWHCP7H33VTCZQBWIK72DIW25M5YXCTRGVSPJI75XSQ) context deadline exceeded ======LOGS====== Dec 22 17:37:54.324416701 (410h8m54.195499737s) [upload=360074]: Starting upload Dec 22 17:37:54.324501634 (410h8m54.195584679s) [upload=360074]: Done starting upload Dec 22 17:37:54.324730208 (410h8m54.195813253s) [upload=360074]: Uploading segments... Dec 22 17:37:54.324896816 (410h8m54.195979861s) [upload=360074]: Got next segment. Inline: false Dec 22 17:37:54.324956119 (410h8m54.196039154s) [upload=360074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:37:54.324962060 (410h8m54.196045105s) [upload=360074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:37:54.571919378 (410h8m54.443002423s) [seg_pos={0 0}, upload=360074]: Begin upload segment... Dec 22 17:37:54.571993309 (410h8m54.443076354s) [seg_pos={0 0}, upload=360074]: Done begin upload segment. Dec 22 17:37:54.572023747 (410h8m54.443106792s) [noise=true , address=145.79.227.38:4978 , piece_id=KQ5UDQZRZ4V4GHANA2X2IOIXSZ7GMAFGNZOKPDNNZ62NIEI3KV4A, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.572038245 (410h8m54.44312128s) [noise=true , address=145.79.227.37:4565 , piece_id=CDD6ON4W3WTVWWIJYROZHF5Q7VE5KQX2M6STFLTKUD3Z3I3L7AWQ, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.572529601 (410h8m54.443612636s) [noise=true , address=45.91.116.11:4117 , piece_id=SIYZEDKL55H42OMOIP45LKLZAAGO7VC73VLM7TMMMCHI37CF3CBQ, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.572918812 (410h8m54.444001847s) [noise=true , address=145.79.227.46:4603 , piece_id=CDUTVUXEXANYJ4OXHCY7MUJMA47RSN6KJLQLG6EAV6K4AYS77EJA, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.574441882 (410h8m54.445524927s) [noise=true , address=145.79.227.29:4942 , piece_id=GBIIRXHJN5PMT7AFI3FZQI4233ZDM5G4EDD226GVL663MW4HTGHA, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.574919060 (410h8m54.446002105s) [noise=true , address=77.90.31.11:4553 , piece_id=7DEIBFTB227UGILLYPPIUKQWLJ4WLEZVFHSTSWH3QUQ6NCZ5VYYA, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.575860574 (410h8m54.446943609s) [noise=true , address=145.79.227.45:4230 , piece_id=QYN2XARJPHOWRVK24MJ7UOFTAHVX7P3RLBRY43FXAZEFRSMYLZ3A, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.576357810 (410h8m54.447440845s) [noise=true , address=145.79.227.47:4265 , piece_id=YHSKF4CMU3FRMLBZE37XDSL55C564XJXJBXYVJZ3VHOCKV2JGOFA, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.577003409 (410h8m54.448086444s) [noise=true , address=145.79.227.11:4732 , piece_id=7GP2GI7LRHF4CFYQXIEFNP4GN2CWEGRROAALITI6WKV6Z6LFITZA, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.577967666 (410h8m54.449050701s) [noise=true , address=145.79.227.51:4409 , piece_id=YTONMN7KANDBI7IGOCLWZJ4VWSTWGO633UYFW5VWBXB4YNXJRPAA, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.579571139 (410h8m54.450654184s) [noise=true , address=77.90.31.14:4663 , piece_id=JYZ45AOTX76XH325PI3ABYYH6GJ5T6U2GAMRPRTUAWMYRJMJCYGQ, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.581130639 (410h8m54.452213674s) [noise=true , address=45.91.116.12:4228 , piece_id=4NQLBN4CGZWHCP7H33VTCZQBWIK72DIW25M5YXCTRGVSPJI75XSQ, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.582164027 (410h8m54.453247082s) [noise=true , address=145.79.227.22:4509 , piece_id=3SSFGONFELFEKHOWU7SX62XH243YE2CYODWZ2AOC5OYZB6GI3K3Q, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.583432482 (410h8m54.454515527s) [noise=true , address=145.79.227.50:4375 , piece_id=WD4BDNU6G6BLM7W6JGIV3ABF3IBRIA7F74KV7FPHDEDEECVBHT2Q, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.583933996 (410h8m54.455017031s) [noise=true , address=45.91.116.5:4297 , piece_id=4YYAU3SDD3SJGIJRLFF2WNNOX22HRWX3ICK3VCZPZZLNVINAKCAQ, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:54.585083877 (410h8m54.456166912s) [noise=true , address=145.79.227.41:4085 , piece_id=MU274JVCFIQXJ3AMCXJZDSL7GYORBH7QYQUI4L4EU5OFRGDXAJIQ, seg_pos={0 0}, upload=360074]: Uploading piece... Dec 22 17:37:56.455063815 (410h8m56.32614686s) [upload=360074]: Next returned nil segment Dec 22 17:37:56.455102759 (410h8m56.326185794s) [upload=360074]: Waiting for error group managing segments... Dec 22 17:37:56.814034366 (410h8m56.685117411s) [noise=true , address=77.90.31.11:4553 , piece_id=7DEIBFTB227UGILLYPPIUKQWLJ4WLEZVFHSTSWH3QUQ6NCZ5VYYA, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:56.847186984 (410h8m56.718270019s) [noise=true , address=77.90.31.14:4663 , piece_id=JYZ45AOTX76XH325PI3ABYYH6GJ5T6U2GAMRPRTUAWMYRJMJCYGQ, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.188011121 (410h8m57.059094186s) [noise=true , address=45.91.116.5:4297 , piece_id=4YYAU3SDD3SJGIJRLFF2WNNOX22HRWX3ICK3VCZPZZLNVINAKCAQ, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.251950432 (410h8m57.123033477s) [noise=true , address=145.79.227.38:4978 , piece_id=KQ5UDQZRZ4V4GHANA2X2IOIXSZ7GMAFGNZOKPDNNZ62NIEI3KV4A, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.256289474 (410h8m57.127372509s) [noise=true , address=145.79.227.46:4603 , piece_id=CDUTVUXEXANYJ4OXHCY7MUJMA47RSN6KJLQLG6EAV6K4AYS77EJA, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.257786254 (410h8m57.128869299s) [noise=true , address=145.79.227.47:4265 , piece_id=YHSKF4CMU3FRMLBZE37XDSL55C564XJXJBXYVJZ3VHOCKV2JGOFA, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.259551064 (410h8m57.130634099s) [noise=true , address=145.79.227.50:4375 , piece_id=WD4BDNU6G6BLM7W6JGIV3ABF3IBRIA7F74KV7FPHDEDEECVBHT2Q, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.519933327 (410h8m57.391016362s) [noise=true , address=45.91.116.11:4117 , piece_id=SIYZEDKL55H42OMOIP45LKLZAAGO7VC73VLM7TMMMCHI37CF3CBQ, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.535713425 (410h8m57.40679646s) [noise=true , address=145.79.227.45:4230 , piece_id=QYN2XARJPHOWRVK24MJ7UOFTAHVX7P3RLBRY43FXAZEFRSMYLZ3A, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.539029579 (410h8m57.410112624s) [noise=true , address=145.79.227.41:4085 , piece_id=MU274JVCFIQXJ3AMCXJZDSL7GYORBH7QYQUI4L4EU5OFRGDXAJIQ, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.551322707 (410h8m57.422405742s) [noise=true , address=145.79.227.37:4565 , piece_id=CDD6ON4W3WTVWWIJYROZHF5Q7VE5KQX2M6STFLTKUD3Z3I3L7AWQ, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.578504316 (410h8m57.449587351s) [noise=true , address=45.91.116.12:4228 , piece_id=4NQLBN4CGZWHCP7H33VTCZQBWIK72DIW25M5YXCTRGVSPJI75XSQ, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.580985600 (410h8m57.452068635s) [noise=true , address=145.79.227.22:4509 , piece_id=3SSFGONFELFEKHOWU7SX62XH243YE2CYODWZ2AOC5OYZB6GI3K3Q, seg_pos={0 0}, upload=360074]: Done uploading piece. err: <nil> Dec 22 17:37:57.581005979 (410h8m57.452089024s) [seg_pos={0 0}, upload=360074]: Segment reached optimal threshold of 13 pieces. Dec 22 17:37:57.581224986 (410h8m57.452308021s) [noise=true , address=145.79.227.51:4409 , piece_id=YTONMN7KANDBI7IGOCLWZJ4VWSTWGO633UYFW5VWBXB4YNXJRPAA, seg_pos={0 0}, upload=360074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 22 17:37:57.581233021 (410h8m57.452316066s) [noise=true , address=145.79.227.51:4409 , piece_id=YTONMN7KANDBI7IGOCLWZJ4VWSTWGO633UYFW5VWBXB4YNXJRPAA, seg_pos={0 0}, upload=360074]: Long tail cancellation due to optimal threshold Dec 22 17:37:57.581518625 (410h8m57.45260167s) [noise=true , address=145.79.227.29:4942 , piece_id=GBIIRXHJN5PMT7AFI3FZQI4233ZDM5G4EDD226GVL663MW4HTGHA, seg_pos={0 0}, upload=360074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 22 17:37:57.581526380 (410h8m57.452609425s) [noise=true , address=145.79.227.29:4942 , piece_id=GBIIRXHJN5PMT7AFI3FZQI4233ZDM5G4EDD226GVL663MW4HTGHA, seg_pos={0 0}, upload=360074]: Long tail cancellation due to optimal threshold Dec 22 17:37:57.581692576 (410h8m57.452775621s) [noise=true , address=145.79.227.11:4732 , piece_id=7GP2GI7LRHF4CFYQXIEFNP4GN2CWEGRROAALITI6WKV6Z6LFITZA, seg_pos={0 0}, upload=360074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Dec 22 17:37:57.581700993 (410h8m57.452784028s) [noise=true , address=145.79.227.11:4732 , piece_id=7GP2GI7LRHF4CFYQXIEFNP4GN2CWEGRROAALITI6WKV6Z6LFITZA, seg_pos={0 0}, upload=360074]: Long tail cancellation due to optimal threshold Dec 22 17:37:57.581720259 (410h8m57.452803304s) [seg_pos={0 0}, upload=360074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:37:57.581840829 (410h8m57.452923874s) [upload=360074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:37:57.581847011 (410h8m57.452930056s) [upload=360074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:37:57.877335941 (410h8m57.748418986s) [upload=360074]: Done uploading segments... Dec 22 17:37:57.877358845 (410h8m57.74844188s) [upload=360074]: Upload finished. err: <nil> Duration: 30.002265347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X...X...X.....X.........X...........X..X..XX....X...X.X..X..] duration: 19.381651924s trace: https://jaeger.dev.storj.io/trace/6b74221c68a21808 FAIL (2025-12-22 17:37:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 17:35:27.360615331 (410h6m27.231698367s) [upload=360037]: Starting upload Dec 22 17:35:27.360702878 (410h6m27.231785923s) [upload=360037]: Done starting upload Dec 22 17:35:27.360782450 (410h6m27.231865495s) [upload=360037]: Uploading segments... Dec 22 17:35:27.361009462 (410h6m27.232092497s) [upload=360037]: Got next segment. Inline: false Dec 22 17:35:27.361043476 (410h6m27.232126522s) [upload=360037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:35:27.361048566 (410h6m27.232131601s) [upload=360037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:35:27.688662800 (410h6m27.559745845s) [seg_pos={0 0}, upload=360037]: Begin upload segment... Dec 22 17:35:27.688705992 (410h6m27.559789027s) [seg_pos={0 0}, upload=360037]: Done begin upload segment. Dec 22 17:35:27.688732603 (410h6m27.559815638s) [noise=true , address=145.79.227.41:4085 , piece_id=XHZV5CTNPXXAW3U6R3JQBDZ3TALPYUJHHS2KFQPCFB3P3D43Q4WQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.688837553 (410h6m27.559920588s) [noise=true , address=45.91.116.11:4117 , piece_id=GK3YBIPZO3IHAGRARYWDHGQ3A3752V3YV36PO6KSTGVXKTNYT4ZA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.689372942 (410h6m27.560455977s) [noise=true , address=145.79.227.29:4943 , piece_id=M4TTQJMEZ65NSQKD7ZGFZ6MPLYASU6DKOORIFYLGWPOYZA4A3UWQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.689744820 (410h6m27.560827865s) [noise=true , address=145.79.227.11:4732 , piece_id=C4Y7GELSWRECYICEUHTQLF7GZBCKAI3CD2IDIPBHKE5NK4ZYM7TA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.690137588 (410h6m27.561220623s) [noise=true , address=145.79.227.43:4158 , piece_id=2NKPPEFSXQLFYDNZ2TPKNNQDSFK5IGEAR32HFJM5ASLN6UJZN76A, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.690544292 (410h6m27.561627327s) [noise=true , address=145.79.227.34:4907 , piece_id=LXFCFDVT3CLI3JKZKKR3QRNGJLRDIDV56ZF7M62CI557QGWLJF2A, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.690888027 (410h6m27.561971072s) [noise=true , address=145.79.227.44:4195 , piece_id=4LXFB54MTKVOQAU3RQGGSTTOEFK6NHRZQNJ6GTX2V2VYWMEYXXRA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.691234106 (410h6m27.562317151s) [noise=true , address=145.79.227.38:4979 , piece_id=N2F7BAUAWVXHLTMGDUOLRON3EMLMITS74B2A2KQ3XVIMB2OSA2UQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.692625496 (410h6m27.563708541s) [noise=true , address=45.91.116.9:4262 , piece_id=D2SOXDDODTSAYLLJZUCFQ65X7JMAAEAJ23AXSGITXL2YL462TVZA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.692644952 (410h6m27.563727997s) [noise=true , address=77.90.31.17:4773 , piece_id=UWNHJW35VEY5JUJV5BHLF6TV3KGZ4ROCCUCBCKQ7KZ2E5IXLIZMQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.695545254 (410h6m27.566628289s) [noise=true , address=77.90.31.16:4736 , piece_id=GMENHC2GLBUPFV5VBZMKCEKXCR5A556PJ66MS72H5WP3LCQBRN3Q, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.696890767 (410h6m27.567973812s) [noise=true , address=145.79.227.37:4567 , piece_id=UP4USOBUEU7DCH6P7ETNRM5XZLERDRU72SCU2SJJTFRBI2ADMA3Q, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.698121992 (410h6m27.569205027s) [noise=true , address=145.79.227.52:4447 , piece_id=OJTEJQMCKP2LD27QLLSG5WHL32GYBXSS3YMY7AJG4AYMBRHOW4HQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.699453387 (410h6m27.570536432s) [noise=true , address=77.90.31.15:4697 , piece_id=BPVDZLK665WE5RRMKISFT2ZM6IBUKYVP6EVAXP63QDZOPZII5YRQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.701059164 (410h6m27.5721422s) [noise=true , address=145.79.227.45:4230 , piece_id=SZPPXI4GXHFJF7ABBV7F6RIDJH6IMTND5FHW5HTZDNPVIZV7KVGA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.702501300 (410h6m27.573584335s) [noise=true , address=145.79.227.47:4265 , piece_id=2UZMPP5VCRPNQTE6VPSHTRHXP5TZE4CDOLO4VA5AZR37PJDNWDNA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:28.835191945 (410h6m28.70627498s) [noise=true , address=77.90.31.15:4697 , piece_id=BPVDZLK665WE5RRMKISFT2ZM6IBUKYVP6EVAXP63QDZOPZII5YRQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 22 17:35:29.575049930 (410h6m29.446132965s) [upload=360037]: Next returned nil segment Dec 22 17:35:29.575064738 (410h6m29.446147784s) [upload=360037]: Waiting for error group managing segments... Dec 22 17:35:29.875549252 (410h6m29.746632297s) [noise=true , address=77.90.31.17:4773 , piece_id=UWNHJW35VEY5JUJV5BHLF6TV3KGZ4ROCCUCBCKQ7KZ2E5IXLIZMQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:29.959602143 (410h6m29.830685178s) [noise=true , address=45.91.116.9:4262 , piece_id=D2SOXDDODTSAYLLJZUCFQ65X7JMAAEAJ23AXSGITXL2YL462TVZA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.363515663 (410h6m30.234598698s) [noise=true , address=45.91.116.11:4117 , piece_id=GK3YBIPZO3IHAGRARYWDHGQ3A3752V3YV36PO6KSTGVXKTNYT4ZA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.510598193 (410h6m30.381681228s) [noise=true , address=145.79.227.37:4567 , piece_id=UP4USOBUEU7DCH6P7ETNRM5XZLERDRU72SCU2SJJTFRBI2ADMA3Q, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.511603857 (410h6m30.382686892s) [noise=true , address=145.79.227.47:4265 , piece_id=2UZMPP5VCRPNQTE6VPSHTRHXP5TZE4CDOLO4VA5AZR37PJDNWDNA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.513245864 (410h6m30.384328899s) [noise=true , address=145.79.227.52:4447 , piece_id=OJTEJQMCKP2LD27QLLSG5WHL32GYBXSS3YMY7AJG4AYMBRHOW4HQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.514766169 (410h6m30.385849204s) [noise=true , address=145.79.227.38:4979 , piece_id=N2F7BAUAWVXHLTMGDUOLRON3EMLMITS74B2A2KQ3XVIMB2OSA2UQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.533770246 (410h6m30.404853281s) [noise=true , address=145.79.227.43:4158 , piece_id=2NKPPEFSXQLFYDNZ2TPKNNQDSFK5IGEAR32HFJM5ASLN6UJZN76A, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.566208504 (410h6m30.437291539s) [noise=true , address=145.79.227.11:4732 , piece_id=C4Y7GELSWRECYICEUHTQLF7GZBCKAI3CD2IDIPBHKE5NK4ZYM7TA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.578778880 (410h6m30.449861915s) [noise=true , address=145.79.227.34:4907 , piece_id=LXFCFDVT3CLI3JKZKKR3QRNGJLRDIDV56ZF7M62CI557QGWLJF2A, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.663405323 (410h6m30.534488358s) [noise=true , address=77.90.31.16:4736 , piece_id=GMENHC2GLBUPFV5VBZMKCEKXCR5A556PJ66MS72H5WP3LCQBRN3Q, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.707954993 (410h6m30.579038038s) [noise=true , address=145.79.227.41:4085 , piece_id=XHZV5CTNPXXAW3U6R3JQBDZ3TALPYUJHHS2KFQPCFB3P3D43Q4WQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.722481925 (410h6m30.59356496s) [noise=true , address=145.79.227.45:4230 , piece_id=SZPPXI4GXHFJF7ABBV7F6RIDJH6IMTND5FHW5HTZDNPVIZV7KVGA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.722501903 (410h6m30.593584938s) [seg_pos={0 0}, upload=360037]: Segment reached optimal threshold of 13 pieces. Dec 22 17:35:30.722665064 (410h6m30.593748099s) [noise=true , address=145.79.227.29:4943 , piece_id=M4TTQJMEZ65NSQKD7ZGFZ6MPLYASU6DKOORIFYLGWPOYZA4A3UWQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 22 17:35:30.722672718 (410h6m30.593755753s) [noise=true , address=145.79.227.29:4943 , piece_id=M4TTQJMEZ65NSQKD7ZGFZ6MPLYASU6DKOORIFYLGWPOYZA4A3UWQ, seg_pos={0 0}, upload=360037]: Long tail cancellation due to optimal threshold Dec 22 17:35:30.722899320 (410h6m30.593982365s) [noise=true , address=145.79.227.44:4195 , piece_id=4LXFB54MTKVOQAU3RQGGSTTOEFK6NHRZQNJ6GTX2V2VYWMEYXXRA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 22 17:35:30.722905692 (410h6m30.593988727s) [noise=true , address=145.79.227.44:4195 , piece_id=4LXFB54MTKVOQAU3RQGGSTTOEFK6NHRZQNJ6GTX2V2VYWMEYXXRA, seg_pos={0 0}, upload=360037]: Long tail cancellation due to optimal threshold Dec 22 17:35:30.722925329 (410h6m30.594008364s) [seg_pos={0 0}, upload=360037]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:35:30.722989231 (410h6m30.594072276s) [upload=360037]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:35:30.722994461 (410h6m30.594077496s) [upload=360037]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:35:31.109690014 (410h6m30.980773059s) [upload=360037]: Done uploading segments... Dec 22 17:35:31.109711285 (410h6m30.98079433s) [upload=360037]: Upload finished. err: <nil> Duration: 19.381677412s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X...X...X.....X.........X...........X..X..XX....X...X.X..X.] duration: 14.545752114s trace: https://jaeger.dev.storj.io/trace/6b74221c68a21808 FAIL (2025-12-22 17:36:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 17:35:27.360615331 (410h6m27.231698367s) [upload=360037]: Starting upload Dec 22 17:35:27.360702878 (410h6m27.231785923s) [upload=360037]: Done starting upload Dec 22 17:35:27.360782450 (410h6m27.231865495s) [upload=360037]: Uploading segments... Dec 22 17:35:27.361009462 (410h6m27.232092497s) [upload=360037]: Got next segment. Inline: false Dec 22 17:35:27.361043476 (410h6m27.232126522s) [upload=360037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:35:27.361048566 (410h6m27.232131601s) [upload=360037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:35:27.688662800 (410h6m27.559745845s) [seg_pos={0 0}, upload=360037]: Begin upload segment... Dec 22 17:35:27.688705992 (410h6m27.559789027s) [seg_pos={0 0}, upload=360037]: Done begin upload segment. Dec 22 17:35:27.688732603 (410h6m27.559815638s) [noise=true , address=145.79.227.41:4085 , piece_id=XHZV5CTNPXXAW3U6R3JQBDZ3TALPYUJHHS2KFQPCFB3P3D43Q4WQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.688837553 (410h6m27.559920588s) [noise=true , address=45.91.116.11:4117 , piece_id=GK3YBIPZO3IHAGRARYWDHGQ3A3752V3YV36PO6KSTGVXKTNYT4ZA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.689372942 (410h6m27.560455977s) [noise=true , address=145.79.227.29:4943 , piece_id=M4TTQJMEZ65NSQKD7ZGFZ6MPLYASU6DKOORIFYLGWPOYZA4A3UWQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.689744820 (410h6m27.560827865s) [noise=true , address=145.79.227.11:4732 , piece_id=C4Y7GELSWRECYICEUHTQLF7GZBCKAI3CD2IDIPBHKE5NK4ZYM7TA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.690137588 (410h6m27.561220623s) [noise=true , address=145.79.227.43:4158 , piece_id=2NKPPEFSXQLFYDNZ2TPKNNQDSFK5IGEAR32HFJM5ASLN6UJZN76A, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.690544292 (410h6m27.561627327s) [noise=true , address=145.79.227.34:4907 , piece_id=LXFCFDVT3CLI3JKZKKR3QRNGJLRDIDV56ZF7M62CI557QGWLJF2A, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.690888027 (410h6m27.561971072s) [noise=true , address=145.79.227.44:4195 , piece_id=4LXFB54MTKVOQAU3RQGGSTTOEFK6NHRZQNJ6GTX2V2VYWMEYXXRA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.691234106 (410h6m27.562317151s) [noise=true , address=145.79.227.38:4979 , piece_id=N2F7BAUAWVXHLTMGDUOLRON3EMLMITS74B2A2KQ3XVIMB2OSA2UQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.692625496 (410h6m27.563708541s) [noise=true , address=45.91.116.9:4262 , piece_id=D2SOXDDODTSAYLLJZUCFQ65X7JMAAEAJ23AXSGITXL2YL462TVZA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.692644952 (410h6m27.563727997s) [noise=true , address=77.90.31.17:4773 , piece_id=UWNHJW35VEY5JUJV5BHLF6TV3KGZ4ROCCUCBCKQ7KZ2E5IXLIZMQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.695545254 (410h6m27.566628289s) [noise=true , address=77.90.31.16:4736 , piece_id=GMENHC2GLBUPFV5VBZMKCEKXCR5A556PJ66MS72H5WP3LCQBRN3Q, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.696890767 (410h6m27.567973812s) [noise=true , address=145.79.227.37:4567 , piece_id=UP4USOBUEU7DCH6P7ETNRM5XZLERDRU72SCU2SJJTFRBI2ADMA3Q, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.698121992 (410h6m27.569205027s) [noise=true , address=145.79.227.52:4447 , piece_id=OJTEJQMCKP2LD27QLLSG5WHL32GYBXSS3YMY7AJG4AYMBRHOW4HQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.699453387 (410h6m27.570536432s) [noise=true , address=77.90.31.15:4697 , piece_id=BPVDZLK665WE5RRMKISFT2ZM6IBUKYVP6EVAXP63QDZOPZII5YRQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.701059164 (410h6m27.5721422s) [noise=true , address=145.79.227.45:4230 , piece_id=SZPPXI4GXHFJF7ABBV7F6RIDJH6IMTND5FHW5HTZDNPVIZV7KVGA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.702501300 (410h6m27.573584335s) [noise=true , address=145.79.227.47:4265 , piece_id=2UZMPP5VCRPNQTE6VPSHTRHXP5TZE4CDOLO4VA5AZR37PJDNWDNA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:28.835191945 (410h6m28.70627498s) [noise=true , address=77.90.31.15:4697 , piece_id=BPVDZLK665WE5RRMKISFT2ZM6IBUKYVP6EVAXP63QDZOPZII5YRQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 22 17:35:29.575049930 (410h6m29.446132965s) [upload=360037]: Next returned nil segment Dec 22 17:35:29.575064738 (410h6m29.446147784s) [upload=360037]: Waiting for error group managing segments... Dec 22 17:35:29.875549252 (410h6m29.746632297s) [noise=true , address=77.90.31.17:4773 , piece_id=UWNHJW35VEY5JUJV5BHLF6TV3KGZ4ROCCUCBCKQ7KZ2E5IXLIZMQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:29.959602143 (410h6m29.830685178s) [noise=true , address=45.91.116.9:4262 , piece_id=D2SOXDDODTSAYLLJZUCFQ65X7JMAAEAJ23AXSGITXL2YL462TVZA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.363515663 (410h6m30.234598698s) [noise=true , address=45.91.116.11:4117 , piece_id=GK3YBIPZO3IHAGRARYWDHGQ3A3752V3YV36PO6KSTGVXKTNYT4ZA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.510598193 (410h6m30.381681228s) [noise=true , address=145.79.227.37:4567 , piece_id=UP4USOBUEU7DCH6P7ETNRM5XZLERDRU72SCU2SJJTFRBI2ADMA3Q, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.511603857 (410h6m30.382686892s) [noise=true , address=145.79.227.47:4265 , piece_id=2UZMPP5VCRPNQTE6VPSHTRHXP5TZE4CDOLO4VA5AZR37PJDNWDNA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.513245864 (410h6m30.384328899s) [noise=true , address=145.79.227.52:4447 , piece_id=OJTEJQMCKP2LD27QLLSG5WHL32GYBXSS3YMY7AJG4AYMBRHOW4HQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.514766169 (410h6m30.385849204s) [noise=true , address=145.79.227.38:4979 , piece_id=N2F7BAUAWVXHLTMGDUOLRON3EMLMITS74B2A2KQ3XVIMB2OSA2UQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.533770246 (410h6m30.404853281s) [noise=true , address=145.79.227.43:4158 , piece_id=2NKPPEFSXQLFYDNZ2TPKNNQDSFK5IGEAR32HFJM5ASLN6UJZN76A, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.566208504 (410h6m30.437291539s) [noise=true , address=145.79.227.11:4732 , piece_id=C4Y7GELSWRECYICEUHTQLF7GZBCKAI3CD2IDIPBHKE5NK4ZYM7TA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.578778880 (410h6m30.449861915s) [noise=true , address=145.79.227.34:4907 , piece_id=LXFCFDVT3CLI3JKZKKR3QRNGJLRDIDV56ZF7M62CI557QGWLJF2A, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.663405323 (410h6m30.534488358s) [noise=true , address=77.90.31.16:4736 , piece_id=GMENHC2GLBUPFV5VBZMKCEKXCR5A556PJ66MS72H5WP3LCQBRN3Q, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.707954993 (410h6m30.579038038s) [noise=true , address=145.79.227.41:4085 , piece_id=XHZV5CTNPXXAW3U6R3JQBDZ3TALPYUJHHS2KFQPCFB3P3D43Q4WQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.722481925 (410h6m30.59356496s) [noise=true , address=145.79.227.45:4230 , piece_id=SZPPXI4GXHFJF7ABBV7F6RIDJH6IMTND5FHW5HTZDNPVIZV7KVGA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.722501903 (410h6m30.593584938s) [seg_pos={0 0}, upload=360037]: Segment reached optimal threshold of 13 pieces. Dec 22 17:35:30.722665064 (410h6m30.593748099s) [noise=true , address=145.79.227.29:4943 , piece_id=M4TTQJMEZ65NSQKD7ZGFZ6MPLYASU6DKOORIFYLGWPOYZA4A3UWQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 22 17:35:30.722672718 (410h6m30.593755753s) [noise=true , address=145.79.227.29:4943 , piece_id=M4TTQJMEZ65NSQKD7ZGFZ6MPLYASU6DKOORIFYLGWPOYZA4A3UWQ, seg_pos={0 0}, upload=360037]: Long tail cancellation due to optimal threshold Dec 22 17:35:30.722899320 (410h6m30.593982365s) [noise=true , address=145.79.227.44:4195 , piece_id=4LXFB54MTKVOQAU3RQGGSTTOEFK6NHRZQNJ6GTX2V2VYWMEYXXRA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 22 17:35:30.722905692 (410h6m30.593988727s) [noise=true , address=145.79.227.44:4195 , piece_id=4LXFB54MTKVOQAU3RQGGSTTOEFK6NHRZQNJ6GTX2V2VYWMEYXXRA, seg_pos={0 0}, upload=360037]: Long tail cancellation due to optimal threshold Dec 22 17:35:30.722925329 (410h6m30.594008364s) [seg_pos={0 0}, upload=360037]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:35:30.722989231 (410h6m30.594072276s) [upload=360037]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:35:30.722994461 (410h6m30.594077496s) [upload=360037]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:35:31.109690014 (410h6m30.980773059s) [upload=360037]: Done uploading segments... Dec 22 17:35:31.109711285 (410h6m30.98079433s) [upload=360037]: Upload finished. err: <nil> Duration: 14.545885518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X...X...X.....X.........X...........X..X..XX....X...X.X..X] duration: 30.003286234s trace: https://jaeger.dev.storj.io/trace/6b74221c68a21808 FAIL (2025-12-22 17:35:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 17:35:27.360615331 (410h6m27.231698367s) [upload=360037]: Starting upload Dec 22 17:35:27.360702878 (410h6m27.231785923s) [upload=360037]: Done starting upload Dec 22 17:35:27.360782450 (410h6m27.231865495s) [upload=360037]: Uploading segments... Dec 22 17:35:27.361009462 (410h6m27.232092497s) [upload=360037]: Got next segment. Inline: false Dec 22 17:35:27.361043476 (410h6m27.232126522s) [upload=360037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:35:27.361048566 (410h6m27.232131601s) [upload=360037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:35:27.688662800 (410h6m27.559745845s) [seg_pos={0 0}, upload=360037]: Begin upload segment... Dec 22 17:35:27.688705992 (410h6m27.559789027s) [seg_pos={0 0}, upload=360037]: Done begin upload segment. Dec 22 17:35:27.688732603 (410h6m27.559815638s) [noise=true , address=145.79.227.41:4085 , piece_id=XHZV5CTNPXXAW3U6R3JQBDZ3TALPYUJHHS2KFQPCFB3P3D43Q4WQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.688837553 (410h6m27.559920588s) [noise=true , address=45.91.116.11:4117 , piece_id=GK3YBIPZO3IHAGRARYWDHGQ3A3752V3YV36PO6KSTGVXKTNYT4ZA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.689372942 (410h6m27.560455977s) [noise=true , address=145.79.227.29:4943 , piece_id=M4TTQJMEZ65NSQKD7ZGFZ6MPLYASU6DKOORIFYLGWPOYZA4A3UWQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.689744820 (410h6m27.560827865s) [noise=true , address=145.79.227.11:4732 , piece_id=C4Y7GELSWRECYICEUHTQLF7GZBCKAI3CD2IDIPBHKE5NK4ZYM7TA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.690137588 (410h6m27.561220623s) [noise=true , address=145.79.227.43:4158 , piece_id=2NKPPEFSXQLFYDNZ2TPKNNQDSFK5IGEAR32HFJM5ASLN6UJZN76A, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.690544292 (410h6m27.561627327s) [noise=true , address=145.79.227.34:4907 , piece_id=LXFCFDVT3CLI3JKZKKR3QRNGJLRDIDV56ZF7M62CI557QGWLJF2A, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.690888027 (410h6m27.561971072s) [noise=true , address=145.79.227.44:4195 , piece_id=4LXFB54MTKVOQAU3RQGGSTTOEFK6NHRZQNJ6GTX2V2VYWMEYXXRA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.691234106 (410h6m27.562317151s) [noise=true , address=145.79.227.38:4979 , piece_id=N2F7BAUAWVXHLTMGDUOLRON3EMLMITS74B2A2KQ3XVIMB2OSA2UQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.692625496 (410h6m27.563708541s) [noise=true , address=45.91.116.9:4262 , piece_id=D2SOXDDODTSAYLLJZUCFQ65X7JMAAEAJ23AXSGITXL2YL462TVZA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.692644952 (410h6m27.563727997s) [noise=true , address=77.90.31.17:4773 , piece_id=UWNHJW35VEY5JUJV5BHLF6TV3KGZ4ROCCUCBCKQ7KZ2E5IXLIZMQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.695545254 (410h6m27.566628289s) [noise=true , address=77.90.31.16:4736 , piece_id=GMENHC2GLBUPFV5VBZMKCEKXCR5A556PJ66MS72H5WP3LCQBRN3Q, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.696890767 (410h6m27.567973812s) [noise=true , address=145.79.227.37:4567 , piece_id=UP4USOBUEU7DCH6P7ETNRM5XZLERDRU72SCU2SJJTFRBI2ADMA3Q, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.698121992 (410h6m27.569205027s) [noise=true , address=145.79.227.52:4447 , piece_id=OJTEJQMCKP2LD27QLLSG5WHL32GYBXSS3YMY7AJG4AYMBRHOW4HQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.699453387 (410h6m27.570536432s) [noise=true , address=77.90.31.15:4697 , piece_id=BPVDZLK665WE5RRMKISFT2ZM6IBUKYVP6EVAXP63QDZOPZII5YRQ, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.701059164 (410h6m27.5721422s) [noise=true , address=145.79.227.45:4230 , piece_id=SZPPXI4GXHFJF7ABBV7F6RIDJH6IMTND5FHW5HTZDNPVIZV7KVGA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:27.702501300 (410h6m27.573584335s) [noise=true , address=145.79.227.47:4265 , piece_id=2UZMPP5VCRPNQTE6VPSHTRHXP5TZE4CDOLO4VA5AZR37PJDNWDNA, seg_pos={0 0}, upload=360037]: Uploading piece... Dec 22 17:35:28.835191945 (410h6m28.70627498s) [noise=true , address=77.90.31.15:4697 , piece_id=BPVDZLK665WE5RRMKISFT2ZM6IBUKYVP6EVAXP63QDZOPZII5YRQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 22 17:35:29.575049930 (410h6m29.446132965s) [upload=360037]: Next returned nil segment Dec 22 17:35:29.575064738 (410h6m29.446147784s) [upload=360037]: Waiting for error group managing segments... Dec 22 17:35:29.875549252 (410h6m29.746632297s) [noise=true , address=77.90.31.17:4773 , piece_id=UWNHJW35VEY5JUJV5BHLF6TV3KGZ4ROCCUCBCKQ7KZ2E5IXLIZMQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:29.959602143 (410h6m29.830685178s) [noise=true , address=45.91.116.9:4262 , piece_id=D2SOXDDODTSAYLLJZUCFQ65X7JMAAEAJ23AXSGITXL2YL462TVZA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.363515663 (410h6m30.234598698s) [noise=true , address=45.91.116.11:4117 , piece_id=GK3YBIPZO3IHAGRARYWDHGQ3A3752V3YV36PO6KSTGVXKTNYT4ZA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.510598193 (410h6m30.381681228s) [noise=true , address=145.79.227.37:4567 , piece_id=UP4USOBUEU7DCH6P7ETNRM5XZLERDRU72SCU2SJJTFRBI2ADMA3Q, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.511603857 (410h6m30.382686892s) [noise=true , address=145.79.227.47:4265 , piece_id=2UZMPP5VCRPNQTE6VPSHTRHXP5TZE4CDOLO4VA5AZR37PJDNWDNA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.513245864 (410h6m30.384328899s) [noise=true , address=145.79.227.52:4447 , piece_id=OJTEJQMCKP2LD27QLLSG5WHL32GYBXSS3YMY7AJG4AYMBRHOW4HQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.514766169 (410h6m30.385849204s) [noise=true , address=145.79.227.38:4979 , piece_id=N2F7BAUAWVXHLTMGDUOLRON3EMLMITS74B2A2KQ3XVIMB2OSA2UQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.533770246 (410h6m30.404853281s) [noise=true , address=145.79.227.43:4158 , piece_id=2NKPPEFSXQLFYDNZ2TPKNNQDSFK5IGEAR32HFJM5ASLN6UJZN76A, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.566208504 (410h6m30.437291539s) [noise=true , address=145.79.227.11:4732 , piece_id=C4Y7GELSWRECYICEUHTQLF7GZBCKAI3CD2IDIPBHKE5NK4ZYM7TA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.578778880 (410h6m30.449861915s) [noise=true , address=145.79.227.34:4907 , piece_id=LXFCFDVT3CLI3JKZKKR3QRNGJLRDIDV56ZF7M62CI557QGWLJF2A, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.663405323 (410h6m30.534488358s) [noise=true , address=77.90.31.16:4736 , piece_id=GMENHC2GLBUPFV5VBZMKCEKXCR5A556PJ66MS72H5WP3LCQBRN3Q, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.707954993 (410h6m30.579038038s) [noise=true , address=145.79.227.41:4085 , piece_id=XHZV5CTNPXXAW3U6R3JQBDZ3TALPYUJHHS2KFQPCFB3P3D43Q4WQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.722481925 (410h6m30.59356496s) [noise=true , address=145.79.227.45:4230 , piece_id=SZPPXI4GXHFJF7ABBV7F6RIDJH6IMTND5FHW5HTZDNPVIZV7KVGA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: <nil> Dec 22 17:35:30.722501903 (410h6m30.593584938s) [seg_pos={0 0}, upload=360037]: Segment reached optimal threshold of 13 pieces. Dec 22 17:35:30.722665064 (410h6m30.593748099s) [noise=true , address=145.79.227.29:4943 , piece_id=M4TTQJMEZ65NSQKD7ZGFZ6MPLYASU6DKOORIFYLGWPOYZA4A3UWQ, seg_pos={0 0}, upload=360037]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 22 17:35:30.722672718 (410h6m30.593755753s) [noise=true , address=145.79.227.29:4943 , piece_id=M4TTQJMEZ65NSQKD7ZGFZ6MPLYASU6DKOORIFYLGWPOYZA4A3UWQ, seg_pos={0 0}, upload=360037]: Long tail cancellation due to optimal threshold Dec 22 17:35:30.722899320 (410h6m30.593982365s) [noise=true , address=145.79.227.44:4195 , piece_id=4LXFB54MTKVOQAU3RQGGSTTOEFK6NHRZQNJ6GTX2V2VYWMEYXXRA, seg_pos={0 0}, upload=360037]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 22 17:35:30.722905692 (410h6m30.593988727s) [noise=true , address=145.79.227.44:4195 , piece_id=4LXFB54MTKVOQAU3RQGGSTTOEFK6NHRZQNJ6GTX2V2VYWMEYXXRA, seg_pos={0 0}, upload=360037]: Long tail cancellation due to optimal threshold Dec 22 17:35:30.722925329 (410h6m30.594008364s) [seg_pos={0 0}, upload=360037]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:35:30.722989231 (410h6m30.594072276s) [upload=360037]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:35:30.722994461 (410h6m30.594077496s) [upload=360037]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:35:31.109690014 (410h6m30.980773059s) [upload=360037]: Done uploading segments... Dec 22 17:35:31.109711285 (410h6m30.98079433s) [upload=360037]: Upload finished. err: <nil> Duration: 30.003311923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X...X...X.....X.........X...........X..X..XX....X...X.X..] duration: 8.98935378s trace: https://jaeger.dev.storj.io/trace/6b2691b5eb97c97c FAIL (2025-12-22 17:35:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: FT3I3YGGSKRVT5YYM2IIUC5SWFPO3FF6KB5L5KAW3YLFCPTS5FTA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: C6XVTQJXLNMHEOLBD3OHPNMGTO4QJZWTIRJLCHHZIOEMGW5R33HA) context deadline exceeded ======LOGS====== Dec 22 17:33:15.006592530 (410h4m14.877675576s) [upload=360006]: Starting upload Dec 22 17:33:15.006688443 (410h4m14.877771498s) [upload=360006]: Done starting upload Dec 22 17:33:15.006743599 (410h4m14.877826664s) [upload=360006]: Uploading segments... Dec 22 17:33:15.006989196 (410h4m14.878072251s) [upload=360006]: Got next segment. Inline: false Dec 22 17:33:15.007025586 (410h4m14.878108631s) [upload=360006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:33:15.007031607 (410h4m14.878114652s) [upload=360006]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:33:15.346937104 (410h4m15.218020149s) [seg_pos={0 0}, upload=360006]: Begin upload segment... Dec 22 17:33:15.346991908 (410h4m15.218074943s) [seg_pos={0 0}, upload=360006]: Done begin upload segment. Dec 22 17:33:15.347018920 (410h4m15.218101955s) [noise=true , address=145.79.227.38:4978 , piece_id=KJMUNZBXLAJPNOCOYAYHVPJJS7TN7K5RKJLVEZSDUAP3Y5ZKHWNA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347051953 (410h4m15.218134988s) [noise=true , address=45.91.116.10:4084 , piece_id=C6XVTQJXLNMHEOLBD3OHPNMGTO4QJZWTIRJLCHHZIOEMGW5R33HA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347389095 (410h4m15.21847214s) [noise=true , address=45.91.116.4:4939 , piece_id=HBRB2F3G2G5N6KMLABKO5WULWLB4OK5DR5CYHGR2CVNZL46KR52Q, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347660761 (410h4m15.218743797s) [noise=true , address=145.79.227.36:4519 , piece_id=FK273VKOMVO6ZWH2DC7ZFBGOQXO5PRSZU6QOL7RUPBVCSXCKS3AQ, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347933922 (410h4m15.219016967s) [noise=true , address=145.79.227.8:4330 , piece_id=VXJHPLCBTAEZY36GRB6HRSKTWPPDZYNLBDGQLWSWUNRY247XPPIA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348279269 (410h4m15.219362305s) [noise=true , address=145.79.227.37:4565 , piece_id=DKWXIGTYUF3E2QF5ISJYBBQVWVQLITREMFQIASXLS6MRZU7HPRHQ, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348546619 (410h4m15.219629654s) [noise=true , address=145.79.227.28:4726 , piece_id=OGM3WFPZDQ3YZSRUKJ7SH464BAZINZTFJO3THSEADXDMFUBYLWZA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348822975 (410h4m15.21990601s) [noise=true , address=145.79.227.39:4015 , piece_id=J6FEZWKTOWFYOKRINVZIWIFJG2LK563OFEE6KMQN7MY36MKYV44A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348833395 (410h4m15.21991643s) [noise=true , address=145.79.227.7:4294 , piece_id=YXKOCMM37VNJREZLFP5BETR6K7W2WNHJNZMSNI72EC5ATPSBYG7A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.349192840 (410h4m15.220275875s) [noise=true , address=77.90.31.14:4663 , piece_id=FT3I3YGGSKRVT5YYM2IIUC5SWFPO3FF6KB5L5KAW3YLFCPTS5FTA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.349531454 (410h4m15.220614499s) [noise=true , address=145.79.227.14:4840 , piece_id=CBYD46LH2CDBVDTAXZCCBYTHEUPZSGP2QDPLFJH2MROMTNXLOM2Q, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.349848167 (410h4m15.220931202s) [noise=true , address=145.79.227.13:4803 , piece_id=XPDJDHPWKH3RGYJ5N6WNDWO3BVEVJRXNOWENUN5LX36MNDS6NWKA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.350167466 (410h4m15.221250511s) [noise=true , address=45.91.116.9:4264 , piece_id=5T6R6U53AG7D6YJE22QDYSLDIMEYYL2FJIPW75A6ZJJXWY4BV27A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.350529304 (410h4m15.221612349s) [noise=true , address=145.79.227.49:4338 , piece_id=25LBCSDZAYDBUIL2BBTNQPAFST3YGXAB73ZAIRGY4CYHLYTUWH7Q, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.350871476 (410h4m15.221954511s) [noise=true , address=145.79.227.51:4409 , piece_id=2DTKWDSV4XZVGYBTEHBKV24JQC6PCTO7QIUTKICEMWGWFHFGK6YA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.356621966 (410h4m15.227705001s) [noise=true , address=145.79.227.33:4871 , piece_id=MCU2HJIPRBZGBEWV7IAEOETXWLMGKLBWM5WBYOQI6ULAHNV5PO6A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:17.251194542 (410h4m17.122277577s) [upload=360006]: Next returned nil segment Dec 22 17:33:17.251215282 (410h4m17.122298317s) [upload=360006]: Waiting for error group managing segments... Dec 22 17:33:17.551209662 (410h4m17.422292697s) [noise=true , address=77.90.31.14:4663 , piece_id=FT3I3YGGSKRVT5YYM2IIUC5SWFPO3FF6KB5L5KAW3YLFCPTS5FTA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:17.604008059 (410h4m17.475091104s) [noise=true , address=45.91.116.4:4939 , piece_id=HBRB2F3G2G5N6KMLABKO5WULWLB4OK5DR5CYHGR2CVNZL46KR52Q, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:17.677634707 (410h4m17.548717752s) [noise=true , address=45.91.116.9:4264 , piece_id=5T6R6U53AG7D6YJE22QDYSLDIMEYYL2FJIPW75A6ZJJXWY4BV27A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:17.907584159 (410h4m17.778667194s) [noise=true , address=45.91.116.10:4084 , piece_id=C6XVTQJXLNMHEOLBD3OHPNMGTO4QJZWTIRJLCHHZIOEMGW5R33HA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.185388256 (410h4m18.056471291s) [noise=true , address=145.79.227.36:4519 , piece_id=FK273VKOMVO6ZWH2DC7ZFBGOQXO5PRSZU6QOL7RUPBVCSXCKS3AQ, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.193982750 (410h4m18.065065785s) [noise=true , address=145.79.227.39:4015 , piece_id=J6FEZWKTOWFYOKRINVZIWIFJG2LK563OFEE6KMQN7MY36MKYV44A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.196433437 (410h4m18.067516483s) [noise=true , address=145.79.227.49:4338 , piece_id=25LBCSDZAYDBUIL2BBTNQPAFST3YGXAB73ZAIRGY4CYHLYTUWH7Q, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.238806012 (410h4m18.109889047s) [noise=true , address=145.79.227.13:4803 , piece_id=XPDJDHPWKH3RGYJ5N6WNDWO3BVEVJRXNOWENUN5LX36MNDS6NWKA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.242960302 (410h4m18.114043337s) [noise=true , address=145.79.227.37:4565 , piece_id=DKWXIGTYUF3E2QF5ISJYBBQVWVQLITREMFQIASXLS6MRZU7HPRHQ, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.249885509 (410h4m18.120968544s) [noise=true , address=145.79.227.33:4871 , piece_id=MCU2HJIPRBZGBEWV7IAEOETXWLMGKLBWM5WBYOQI6ULAHNV5PO6A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.257426388 (410h4m18.128509423s) [noise=true , address=145.79.227.28:4726 , piece_id=OGM3WFPZDQ3YZSRUKJ7SH464BAZINZTFJO3THSEADXDMFUBYLWZA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.258078299 (410h4m18.129161334s) [noise=true , address=145.79.227.51:4409 , piece_id=2DTKWDSV4XZVGYBTEHBKV24JQC6PCTO7QIUTKICEMWGWFHFGK6YA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.278678055 (410h4m18.14976109s) [noise=true , address=145.79.227.38:4978 , piece_id=KJMUNZBXLAJPNOCOYAYHVPJJS7TN7K5RKJLVEZSDUAP3Y5ZKHWNA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.278700798 (410h4m18.149783854s) [seg_pos={0 0}, upload=360006]: Segment reached optimal threshold of 13 pieces. Dec 22 17:33:18.278893595 (410h4m18.14997664s) [noise=true , address=145.79.227.14:4840 , piece_id=CBYD46LH2CDBVDTAXZCCBYTHEUPZSGP2QDPLFJH2MROMTNXLOM2Q, seg_pos={0 0}, upload=360006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Dec 22 17:33:18.278900609 (410h4m18.149983654s) [noise=true , address=145.79.227.14:4840 , piece_id=CBYD46LH2CDBVDTAXZCCBYTHEUPZSGP2QDPLFJH2MROMTNXLOM2Q, seg_pos={0 0}, upload=360006]: Long tail cancellation due to optimal threshold Dec 22 17:33:18.279119776 (410h4m18.150202811s) [noise=true , address=145.79.227.7:4294 , piece_id=YXKOCMM37VNJREZLFP5BETR6K7W2WNHJNZMSNI72EC5ATPSBYG7A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Dec 22 17:33:18.279126399 (410h4m18.150209434s) [noise=true , address=145.79.227.7:4294 , piece_id=YXKOCMM37VNJREZLFP5BETR6K7W2WNHJNZMSNI72EC5ATPSBYG7A, seg_pos={0 0}, upload=360006]: Long tail cancellation due to optimal threshold Dec 22 17:33:18.279142690 (410h4m18.150225735s) [noise=true , address=145.79.227.8:4330 , piece_id=VXJHPLCBTAEZY36GRB6HRSKTWPPDZYNLBDGQLWSWUNRY247XPPIA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Dec 22 17:33:18.279159211 (410h4m18.150242246s) [noise=true , address=145.79.227.8:4330 , piece_id=VXJHPLCBTAEZY36GRB6HRSKTWPPDZYNLBDGQLWSWUNRY247XPPIA, seg_pos={0 0}, upload=360006]: Long tail cancellation due to optimal threshold Dec 22 17:33:18.279187295 (410h4m18.15027033s) [seg_pos={0 0}, upload=360006]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:33:18.279281274 (410h4m18.150364319s) [upload=360006]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:33:18.279288628 (410h4m18.150371663s) [upload=360006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:33:18.687082105 (410h4m18.558165151s) [upload=360006]: Done uploading segments... Dec 22 17:33:18.687101061 (410h4m18.558184096s) [upload=360006]: Upload finished. err: <nil> Duration: 8.989383769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X.X...X...X.....X.........X...........X..X..XX....X...X.X.] duration: 8.697497577s trace: https://jaeger.dev.storj.io/trace/6b2691b5eb97c97c FAIL (2025-12-22 17:34:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: FT3I3YGGSKRVT5YYM2IIUC5SWFPO3FF6KB5L5KAW3YLFCPTS5FTA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: C6XVTQJXLNMHEOLBD3OHPNMGTO4QJZWTIRJLCHHZIOEMGW5R33HA) context deadline exceeded ======LOGS====== Dec 22 17:33:15.006592530 (410h4m14.877675576s) [upload=360006]: Starting upload Dec 22 17:33:15.006688443 (410h4m14.877771498s) [upload=360006]: Done starting upload Dec 22 17:33:15.006743599 (410h4m14.877826664s) [upload=360006]: Uploading segments... Dec 22 17:33:15.006989196 (410h4m14.878072251s) [upload=360006]: Got next segment. Inline: false Dec 22 17:33:15.007025586 (410h4m14.878108631s) [upload=360006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:33:15.007031607 (410h4m14.878114652s) [upload=360006]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:33:15.346937104 (410h4m15.218020149s) [seg_pos={0 0}, upload=360006]: Begin upload segment... Dec 22 17:33:15.346991908 (410h4m15.218074943s) [seg_pos={0 0}, upload=360006]: Done begin upload segment. Dec 22 17:33:15.347018920 (410h4m15.218101955s) [noise=true , address=145.79.227.38:4978 , piece_id=KJMUNZBXLAJPNOCOYAYHVPJJS7TN7K5RKJLVEZSDUAP3Y5ZKHWNA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347051953 (410h4m15.218134988s) [noise=true , address=45.91.116.10:4084 , piece_id=C6XVTQJXLNMHEOLBD3OHPNMGTO4QJZWTIRJLCHHZIOEMGW5R33HA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347389095 (410h4m15.21847214s) [noise=true , address=45.91.116.4:4939 , piece_id=HBRB2F3G2G5N6KMLABKO5WULWLB4OK5DR5CYHGR2CVNZL46KR52Q, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347660761 (410h4m15.218743797s) [noise=true , address=145.79.227.36:4519 , piece_id=FK273VKOMVO6ZWH2DC7ZFBGOQXO5PRSZU6QOL7RUPBVCSXCKS3AQ, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347933922 (410h4m15.219016967s) [noise=true , address=145.79.227.8:4330 , piece_id=VXJHPLCBTAEZY36GRB6HRSKTWPPDZYNLBDGQLWSWUNRY247XPPIA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348279269 (410h4m15.219362305s) [noise=true , address=145.79.227.37:4565 , piece_id=DKWXIGTYUF3E2QF5ISJYBBQVWVQLITREMFQIASXLS6MRZU7HPRHQ, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348546619 (410h4m15.219629654s) [noise=true , address=145.79.227.28:4726 , piece_id=OGM3WFPZDQ3YZSRUKJ7SH464BAZINZTFJO3THSEADXDMFUBYLWZA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348822975 (410h4m15.21990601s) [noise=true , address=145.79.227.39:4015 , piece_id=J6FEZWKTOWFYOKRINVZIWIFJG2LK563OFEE6KMQN7MY36MKYV44A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348833395 (410h4m15.21991643s) [noise=true , address=145.79.227.7:4294 , piece_id=YXKOCMM37VNJREZLFP5BETR6K7W2WNHJNZMSNI72EC5ATPSBYG7A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.349192840 (410h4m15.220275875s) [noise=true , address=77.90.31.14:4663 , piece_id=FT3I3YGGSKRVT5YYM2IIUC5SWFPO3FF6KB5L5KAW3YLFCPTS5FTA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.349531454 (410h4m15.220614499s) [noise=true , address=145.79.227.14:4840 , piece_id=CBYD46LH2CDBVDTAXZCCBYTHEUPZSGP2QDPLFJH2MROMTNXLOM2Q, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.349848167 (410h4m15.220931202s) [noise=true , address=145.79.227.13:4803 , piece_id=XPDJDHPWKH3RGYJ5N6WNDWO3BVEVJRXNOWENUN5LX36MNDS6NWKA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.350167466 (410h4m15.221250511s) [noise=true , address=45.91.116.9:4264 , piece_id=5T6R6U53AG7D6YJE22QDYSLDIMEYYL2FJIPW75A6ZJJXWY4BV27A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.350529304 (410h4m15.221612349s) [noise=true , address=145.79.227.49:4338 , piece_id=25LBCSDZAYDBUIL2BBTNQPAFST3YGXAB73ZAIRGY4CYHLYTUWH7Q, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.350871476 (410h4m15.221954511s) [noise=true , address=145.79.227.51:4409 , piece_id=2DTKWDSV4XZVGYBTEHBKV24JQC6PCTO7QIUTKICEMWGWFHFGK6YA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.356621966 (410h4m15.227705001s) [noise=true , address=145.79.227.33:4871 , piece_id=MCU2HJIPRBZGBEWV7IAEOETXWLMGKLBWM5WBYOQI6ULAHNV5PO6A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:17.251194542 (410h4m17.122277577s) [upload=360006]: Next returned nil segment Dec 22 17:33:17.251215282 (410h4m17.122298317s) [upload=360006]: Waiting for error group managing segments... Dec 22 17:33:17.551209662 (410h4m17.422292697s) [noise=true , address=77.90.31.14:4663 , piece_id=FT3I3YGGSKRVT5YYM2IIUC5SWFPO3FF6KB5L5KAW3YLFCPTS5FTA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:17.604008059 (410h4m17.475091104s) [noise=true , address=45.91.116.4:4939 , piece_id=HBRB2F3G2G5N6KMLABKO5WULWLB4OK5DR5CYHGR2CVNZL46KR52Q, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:17.677634707 (410h4m17.548717752s) [noise=true , address=45.91.116.9:4264 , piece_id=5T6R6U53AG7D6YJE22QDYSLDIMEYYL2FJIPW75A6ZJJXWY4BV27A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:17.907584159 (410h4m17.778667194s) [noise=true , address=45.91.116.10:4084 , piece_id=C6XVTQJXLNMHEOLBD3OHPNMGTO4QJZWTIRJLCHHZIOEMGW5R33HA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.185388256 (410h4m18.056471291s) [noise=true , address=145.79.227.36:4519 , piece_id=FK273VKOMVO6ZWH2DC7ZFBGOQXO5PRSZU6QOL7RUPBVCSXCKS3AQ, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.193982750 (410h4m18.065065785s) [noise=true , address=145.79.227.39:4015 , piece_id=J6FEZWKTOWFYOKRINVZIWIFJG2LK563OFEE6KMQN7MY36MKYV44A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.196433437 (410h4m18.067516483s) [noise=true , address=145.79.227.49:4338 , piece_id=25LBCSDZAYDBUIL2BBTNQPAFST3YGXAB73ZAIRGY4CYHLYTUWH7Q, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.238806012 (410h4m18.109889047s) [noise=true , address=145.79.227.13:4803 , piece_id=XPDJDHPWKH3RGYJ5N6WNDWO3BVEVJRXNOWENUN5LX36MNDS6NWKA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.242960302 (410h4m18.114043337s) [noise=true , address=145.79.227.37:4565 , piece_id=DKWXIGTYUF3E2QF5ISJYBBQVWVQLITREMFQIASXLS6MRZU7HPRHQ, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.249885509 (410h4m18.120968544s) [noise=true , address=145.79.227.33:4871 , piece_id=MCU2HJIPRBZGBEWV7IAEOETXWLMGKLBWM5WBYOQI6ULAHNV5PO6A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.257426388 (410h4m18.128509423s) [noise=true , address=145.79.227.28:4726 , piece_id=OGM3WFPZDQ3YZSRUKJ7SH464BAZINZTFJO3THSEADXDMFUBYLWZA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.258078299 (410h4m18.129161334s) [noise=true , address=145.79.227.51:4409 , piece_id=2DTKWDSV4XZVGYBTEHBKV24JQC6PCTO7QIUTKICEMWGWFHFGK6YA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.278678055 (410h4m18.14976109s) [noise=true , address=145.79.227.38:4978 , piece_id=KJMUNZBXLAJPNOCOYAYHVPJJS7TN7K5RKJLVEZSDUAP3Y5ZKHWNA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.278700798 (410h4m18.149783854s) [seg_pos={0 0}, upload=360006]: Segment reached optimal threshold of 13 pieces. Dec 22 17:33:18.278893595 (410h4m18.14997664s) [noise=true , address=145.79.227.14:4840 , piece_id=CBYD46LH2CDBVDTAXZCCBYTHEUPZSGP2QDPLFJH2MROMTNXLOM2Q, seg_pos={0 0}, upload=360006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Dec 22 17:33:18.278900609 (410h4m18.149983654s) [noise=true , address=145.79.227.14:4840 , piece_id=CBYD46LH2CDBVDTAXZCCBYTHEUPZSGP2QDPLFJH2MROMTNXLOM2Q, seg_pos={0 0}, upload=360006]: Long tail cancellation due to optimal threshold Dec 22 17:33:18.279119776 (410h4m18.150202811s) [noise=true , address=145.79.227.7:4294 , piece_id=YXKOCMM37VNJREZLFP5BETR6K7W2WNHJNZMSNI72EC5ATPSBYG7A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Dec 22 17:33:18.279126399 (410h4m18.150209434s) [noise=true , address=145.79.227.7:4294 , piece_id=YXKOCMM37VNJREZLFP5BETR6K7W2WNHJNZMSNI72EC5ATPSBYG7A, seg_pos={0 0}, upload=360006]: Long tail cancellation due to optimal threshold Dec 22 17:33:18.279142690 (410h4m18.150225735s) [noise=true , address=145.79.227.8:4330 , piece_id=VXJHPLCBTAEZY36GRB6HRSKTWPPDZYNLBDGQLWSWUNRY247XPPIA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Dec 22 17:33:18.279159211 (410h4m18.150242246s) [noise=true , address=145.79.227.8:4330 , piece_id=VXJHPLCBTAEZY36GRB6HRSKTWPPDZYNLBDGQLWSWUNRY247XPPIA, seg_pos={0 0}, upload=360006]: Long tail cancellation due to optimal threshold Dec 22 17:33:18.279187295 (410h4m18.15027033s) [seg_pos={0 0}, upload=360006]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:33:18.279281274 (410h4m18.150364319s) [upload=360006]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:33:18.279288628 (410h4m18.150371663s) [upload=360006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:33:18.687082105 (410h4m18.558165151s) [upload=360006]: Done uploading segments... Dec 22 17:33:18.687101061 (410h4m18.558184096s) [upload=360006]: Upload finished. err: <nil> Duration: 8.697536412s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.X.X...X...X.....X.........X...........X..X..XX....X...X.X] duration: 30.002848251s trace: https://jaeger.dev.storj.io/trace/6b2691b5eb97c97c FAIL (2025-12-22 17:33:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: FT3I3YGGSKRVT5YYM2IIUC5SWFPO3FF6KB5L5KAW3YLFCPTS5FTA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: C6XVTQJXLNMHEOLBD3OHPNMGTO4QJZWTIRJLCHHZIOEMGW5R33HA) context deadline exceeded ======LOGS====== Dec 22 17:33:15.006592530 (410h4m14.877675576s) [upload=360006]: Starting upload Dec 22 17:33:15.006688443 (410h4m14.877771498s) [upload=360006]: Done starting upload Dec 22 17:33:15.006743599 (410h4m14.877826664s) [upload=360006]: Uploading segments... Dec 22 17:33:15.006989196 (410h4m14.878072251s) [upload=360006]: Got next segment. Inline: false Dec 22 17:33:15.007025586 (410h4m14.878108631s) [upload=360006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:33:15.007031607 (410h4m14.878114652s) [upload=360006]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:33:15.346937104 (410h4m15.218020149s) [seg_pos={0 0}, upload=360006]: Begin upload segment... Dec 22 17:33:15.346991908 (410h4m15.218074943s) [seg_pos={0 0}, upload=360006]: Done begin upload segment. Dec 22 17:33:15.347018920 (410h4m15.218101955s) [noise=true , address=145.79.227.38:4978 , piece_id=KJMUNZBXLAJPNOCOYAYHVPJJS7TN7K5RKJLVEZSDUAP3Y5ZKHWNA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347051953 (410h4m15.218134988s) [noise=true , address=45.91.116.10:4084 , piece_id=C6XVTQJXLNMHEOLBD3OHPNMGTO4QJZWTIRJLCHHZIOEMGW5R33HA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347389095 (410h4m15.21847214s) [noise=true , address=45.91.116.4:4939 , piece_id=HBRB2F3G2G5N6KMLABKO5WULWLB4OK5DR5CYHGR2CVNZL46KR52Q, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347660761 (410h4m15.218743797s) [noise=true , address=145.79.227.36:4519 , piece_id=FK273VKOMVO6ZWH2DC7ZFBGOQXO5PRSZU6QOL7RUPBVCSXCKS3AQ, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.347933922 (410h4m15.219016967s) [noise=true , address=145.79.227.8:4330 , piece_id=VXJHPLCBTAEZY36GRB6HRSKTWPPDZYNLBDGQLWSWUNRY247XPPIA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348279269 (410h4m15.219362305s) [noise=true , address=145.79.227.37:4565 , piece_id=DKWXIGTYUF3E2QF5ISJYBBQVWVQLITREMFQIASXLS6MRZU7HPRHQ, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348546619 (410h4m15.219629654s) [noise=true , address=145.79.227.28:4726 , piece_id=OGM3WFPZDQ3YZSRUKJ7SH464BAZINZTFJO3THSEADXDMFUBYLWZA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348822975 (410h4m15.21990601s) [noise=true , address=145.79.227.39:4015 , piece_id=J6FEZWKTOWFYOKRINVZIWIFJG2LK563OFEE6KMQN7MY36MKYV44A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.348833395 (410h4m15.21991643s) [noise=true , address=145.79.227.7:4294 , piece_id=YXKOCMM37VNJREZLFP5BETR6K7W2WNHJNZMSNI72EC5ATPSBYG7A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.349192840 (410h4m15.220275875s) [noise=true , address=77.90.31.14:4663 , piece_id=FT3I3YGGSKRVT5YYM2IIUC5SWFPO3FF6KB5L5KAW3YLFCPTS5FTA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.349531454 (410h4m15.220614499s) [noise=true , address=145.79.227.14:4840 , piece_id=CBYD46LH2CDBVDTAXZCCBYTHEUPZSGP2QDPLFJH2MROMTNXLOM2Q, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.349848167 (410h4m15.220931202s) [noise=true , address=145.79.227.13:4803 , piece_id=XPDJDHPWKH3RGYJ5N6WNDWO3BVEVJRXNOWENUN5LX36MNDS6NWKA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.350167466 (410h4m15.221250511s) [noise=true , address=45.91.116.9:4264 , piece_id=5T6R6U53AG7D6YJE22QDYSLDIMEYYL2FJIPW75A6ZJJXWY4BV27A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.350529304 (410h4m15.221612349s) [noise=true , address=145.79.227.49:4338 , piece_id=25LBCSDZAYDBUIL2BBTNQPAFST3YGXAB73ZAIRGY4CYHLYTUWH7Q, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.350871476 (410h4m15.221954511s) [noise=true , address=145.79.227.51:4409 , piece_id=2DTKWDSV4XZVGYBTEHBKV24JQC6PCTO7QIUTKICEMWGWFHFGK6YA, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:15.356621966 (410h4m15.227705001s) [noise=true , address=145.79.227.33:4871 , piece_id=MCU2HJIPRBZGBEWV7IAEOETXWLMGKLBWM5WBYOQI6ULAHNV5PO6A, seg_pos={0 0}, upload=360006]: Uploading piece... Dec 22 17:33:17.251194542 (410h4m17.122277577s) [upload=360006]: Next returned nil segment Dec 22 17:33:17.251215282 (410h4m17.122298317s) [upload=360006]: Waiting for error group managing segments... Dec 22 17:33:17.551209662 (410h4m17.422292697s) [noise=true , address=77.90.31.14:4663 , piece_id=FT3I3YGGSKRVT5YYM2IIUC5SWFPO3FF6KB5L5KAW3YLFCPTS5FTA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:17.604008059 (410h4m17.475091104s) [noise=true , address=45.91.116.4:4939 , piece_id=HBRB2F3G2G5N6KMLABKO5WULWLB4OK5DR5CYHGR2CVNZL46KR52Q, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:17.677634707 (410h4m17.548717752s) [noise=true , address=45.91.116.9:4264 , piece_id=5T6R6U53AG7D6YJE22QDYSLDIMEYYL2FJIPW75A6ZJJXWY4BV27A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:17.907584159 (410h4m17.778667194s) [noise=true , address=45.91.116.10:4084 , piece_id=C6XVTQJXLNMHEOLBD3OHPNMGTO4QJZWTIRJLCHHZIOEMGW5R33HA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.185388256 (410h4m18.056471291s) [noise=true , address=145.79.227.36:4519 , piece_id=FK273VKOMVO6ZWH2DC7ZFBGOQXO5PRSZU6QOL7RUPBVCSXCKS3AQ, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.193982750 (410h4m18.065065785s) [noise=true , address=145.79.227.39:4015 , piece_id=J6FEZWKTOWFYOKRINVZIWIFJG2LK563OFEE6KMQN7MY36MKYV44A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.196433437 (410h4m18.067516483s) [noise=true , address=145.79.227.49:4338 , piece_id=25LBCSDZAYDBUIL2BBTNQPAFST3YGXAB73ZAIRGY4CYHLYTUWH7Q, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.238806012 (410h4m18.109889047s) [noise=true , address=145.79.227.13:4803 , piece_id=XPDJDHPWKH3RGYJ5N6WNDWO3BVEVJRXNOWENUN5LX36MNDS6NWKA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.242960302 (410h4m18.114043337s) [noise=true , address=145.79.227.37:4565 , piece_id=DKWXIGTYUF3E2QF5ISJYBBQVWVQLITREMFQIASXLS6MRZU7HPRHQ, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.249885509 (410h4m18.120968544s) [noise=true , address=145.79.227.33:4871 , piece_id=MCU2HJIPRBZGBEWV7IAEOETXWLMGKLBWM5WBYOQI6ULAHNV5PO6A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.257426388 (410h4m18.128509423s) [noise=true , address=145.79.227.28:4726 , piece_id=OGM3WFPZDQ3YZSRUKJ7SH464BAZINZTFJO3THSEADXDMFUBYLWZA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.258078299 (410h4m18.129161334s) [noise=true , address=145.79.227.51:4409 , piece_id=2DTKWDSV4XZVGYBTEHBKV24JQC6PCTO7QIUTKICEMWGWFHFGK6YA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.278678055 (410h4m18.14976109s) [noise=true , address=145.79.227.38:4978 , piece_id=KJMUNZBXLAJPNOCOYAYHVPJJS7TN7K5RKJLVEZSDUAP3Y5ZKHWNA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: <nil> Dec 22 17:33:18.278700798 (410h4m18.149783854s) [seg_pos={0 0}, upload=360006]: Segment reached optimal threshold of 13 pieces. Dec 22 17:33:18.278893595 (410h4m18.14997664s) [noise=true , address=145.79.227.14:4840 , piece_id=CBYD46LH2CDBVDTAXZCCBYTHEUPZSGP2QDPLFJH2MROMTNXLOM2Q, seg_pos={0 0}, upload=360006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Dec 22 17:33:18.278900609 (410h4m18.149983654s) [noise=true , address=145.79.227.14:4840 , piece_id=CBYD46LH2CDBVDTAXZCCBYTHEUPZSGP2QDPLFJH2MROMTNXLOM2Q, seg_pos={0 0}, upload=360006]: Long tail cancellation due to optimal threshold Dec 22 17:33:18.279119776 (410h4m18.150202811s) [noise=true , address=145.79.227.7:4294 , piece_id=YXKOCMM37VNJREZLFP5BETR6K7W2WNHJNZMSNI72EC5ATPSBYG7A, seg_pos={0 0}, upload=360006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Dec 22 17:33:18.279126399 (410h4m18.150209434s) [noise=true , address=145.79.227.7:4294 , piece_id=YXKOCMM37VNJREZLFP5BETR6K7W2WNHJNZMSNI72EC5ATPSBYG7A, seg_pos={0 0}, upload=360006]: Long tail cancellation due to optimal threshold Dec 22 17:33:18.279142690 (410h4m18.150225735s) [noise=true , address=145.79.227.8:4330 , piece_id=VXJHPLCBTAEZY36GRB6HRSKTWPPDZYNLBDGQLWSWUNRY247XPPIA, seg_pos={0 0}, upload=360006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Dec 22 17:33:18.279159211 (410h4m18.150242246s) [noise=true , address=145.79.227.8:4330 , piece_id=VXJHPLCBTAEZY36GRB6HRSKTWPPDZYNLBDGQLWSWUNRY247XPPIA, seg_pos={0 0}, upload=360006]: Long tail cancellation due to optimal threshold Dec 22 17:33:18.279187295 (410h4m18.15027033s) [seg_pos={0 0}, upload=360006]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:33:18.279281274 (410h4m18.150364319s) [upload=360006]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:33:18.279288628 (410h4m18.150371663s) [upload=360006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:33:18.687082105 (410h4m18.558165151s) [upload=360006]: Done uploading segments... Dec 22 17:33:18.687101061 (410h4m18.558184096s) [upload=360006]: Upload finished. err: <nil> Duration: 30.002884371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X.X.X...X...X.....X.........X...........X..X..XX....X...X.] duration: 7.933501882s trace: https://jaeger.dev.storj.io/trace/38082a6ba78f52b1 FAIL (2025-12-22 17:32:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OOFZZGZXRX6SOWMORGLUGK7ZIOYW5Q72PPPFYYYISF3VNP3EYXMQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: U25ZF4NLSEI4Z3PCYOZRODCJ7NPRYMZF4GDER7JK4PXB7ZDWJ4BQ) context deadline exceeded ======LOGS====== Dec 22 17:32:09.352567416 (410h3m9.223650451s) [upload=359990]: Starting upload Dec 22 17:32:09.352634664 (410h3m9.223717709s) [upload=359990]: Done starting upload Dec 22 17:32:09.352830957 (410h3m9.223914002s) [upload=359990]: Uploading segments... Dec 22 17:32:09.352944304 (410h3m9.224027349s) [upload=359990]: Got next segment. Inline: false Dec 22 17:32:09.353007814 (410h3m9.224090849s) [upload=359990]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:32:09.353015149 (410h3m9.224098194s) [upload=359990]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:32:09.603456330 (410h3m9.474539375s) [seg_pos={0 0}, upload=359990]: Begin upload segment... Dec 22 17:32:09.603526253 (410h3m9.474609288s) [seg_pos={0 0}, upload=359990]: Done begin upload segment. Dec 22 17:32:09.603575257 (410h3m9.474658292s) [noise=true , address=45.91.116.6:4976 , piece_id=U25ZF4NLSEI4Z3PCYOZRODCJ7NPRYMZF4GDER7JK4PXB7ZDWJ4BQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.603620963 (410h3m9.474704008s) [noise=true , address=145.79.227.35:4481 , piece_id=TPCJMRPQW6B4WDNTEUEBXMJNE3KPM42QGGKPTXWQDVS3EDKMLHJQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.604461373 (410h3m9.475544418s) [noise=true , address=77.90.31.14:4662 , piece_id=QI53J3SAIRUBRQN6L3USYVRT5XGFYJSEOHRQG6F5R4QXKI7KRNHA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.604526648 (410h3m9.475609693s) [noise=true , address=45.91.116.9:4263 , piece_id=5XMACGPWU6SA32MJEDJ2Z2MAXKTX4HUUA7FKRTBE5YUVAAYF42PA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.604858469 (410h3m9.475941504s) [noise=true , address=145.79.227.41:4087 , piece_id=JKP6C37DWT2DZBKHFF7AMMCUABQCG6ILKNHPQU5ZRQPJB6GH4AFA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.605398318 (410h3m9.476481363s) [noise=true , address=77.90.31.11:4555 , piece_id=ZBCJ3GGEXRV3CXLN7MJNPZH4PNVQB2FMINDJRQQDGZ5RDE3WQQ5A, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.605499039 (410h3m9.476582085s) [noise=true , address=145.79.227.36:4518 , piece_id=23OQCYXXQSLHABSRBXZSWBPCT4DEMU5QPR3C6H34BIZBKI6ZODMQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.605943366 (410h3m9.477026401s) [noise=true , address=145.79.227.40:4049 , piece_id=OKMLVX2ZY2PU3O7LTIXYPICW2RN7523FAVVBKWFUYWE27MDVXPQQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.606168995 (410h3m9.47725204s) [noise=true , address=45.91.116.4:4937 , piece_id=H34DHJALOSHZENQFZBRBG7V27QZISAJBBGRRNUPF4B32LJQNOUBA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.606509494 (410h3m9.477592539s) [noise=true , address=145.79.227.8:4329 , piece_id=T2Y7JZ5NIJ5ZO44CU2V5OKV5N6EGTZDD3WQPPPDA4IY3HHU2C7AQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.606801239 (410h3m9.477884274s) [noise=true , address=145.79.227.48:4302 , piece_id=USKG3XLIJX55W4UPB5KDGQZAFIBHKKTLVOSICOJSGTZUFIZAX7HA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.606816128 (410h3m9.477899173s) [noise=true , address=145.79.227.32:4834 , piece_id=ITRX6QU4H2MQBYSXX7IOC7XAK27ZAXYF2ESJWEKBPBDTWG3SIQLA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.607357728 (410h3m9.478440773s) [noise=true , address=145.79.227.51:4409 , piece_id=MFKLLUWY3ZRV3WUCE6F4V54KIWA23NMQV6A47KMBNMCXY3LF6ASQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.607686936 (410h3m9.478769971s) [noise=true , address=45.91.116.5:4297 , piece_id=OOFZZGZXRX6SOWMORGLUGK7ZIOYW5Q72PPPFYYYISF3VNP3EYXMQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.607942712 (410h3m9.479025748s) [noise=true , address=145.79.227.50:4375 , piece_id=N46X46ILGSTVMK4GMDAGK2S34HLQRWH66CVC3IGU5QYRXFZ3A3EQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.608075656 (410h3m9.479158721s) [noise=true , address=145.79.227.14:4840 , piece_id=BROYFHVRS2KNEY7NFMZQZEQXFBYR3INKWL6MS3H3TUJFRIC45TUA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:11.452748346 (410h3m11.323831381s) [upload=359990]: Next returned nil segment Dec 22 17:32:11.452761570 (410h3m11.323844615s) [upload=359990]: Waiting for error group managing segments... Dec 22 17:32:11.874892199 (410h3m11.745975244s) [noise=true , address=45.91.116.4:4937 , piece_id=H34DHJALOSHZENQFZBRBG7V27QZISAJBBGRRNUPF4B32LJQNOUBA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:11.904464128 (410h3m11.775547163s) [noise=true , address=45.91.116.6:4976 , piece_id=U25ZF4NLSEI4Z3PCYOZRODCJ7NPRYMZF4GDER7JK4PXB7ZDWJ4BQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:11.994458362 (410h3m11.865541397s) [noise=true , address=45.91.116.9:4263 , piece_id=5XMACGPWU6SA32MJEDJ2Z2MAXKTX4HUUA7FKRTBE5YUVAAYF42PA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.018110329 (410h3m11.889193364s) [noise=true , address=77.90.31.11:4555 , piece_id=ZBCJ3GGEXRV3CXLN7MJNPZH4PNVQB2FMINDJRQQDGZ5RDE3WQQ5A, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.064672272 (410h3m11.935755317s) [noise=true , address=77.90.31.14:4662 , piece_id=QI53J3SAIRUBRQN6L3USYVRT5XGFYJSEOHRQG6F5R4QXKI7KRNHA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.260653673 (410h3m12.131736718s) [noise=true , address=45.91.116.5:4297 , piece_id=OOFZZGZXRX6SOWMORGLUGK7ZIOYW5Q72PPPFYYYISF3VNP3EYXMQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.380898330 (410h3m12.251981365s) [noise=true , address=145.79.227.41:4087 , piece_id=JKP6C37DWT2DZBKHFF7AMMCUABQCG6ILKNHPQU5ZRQPJB6GH4AFA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.384214434 (410h3m12.255297469s) [noise=true , address=145.79.227.35:4481 , piece_id=TPCJMRPQW6B4WDNTEUEBXMJNE3KPM42QGGKPTXWQDVS3EDKMLHJQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.396183215 (410h3m12.26726626s) [noise=true , address=145.79.227.32:4834 , piece_id=ITRX6QU4H2MQBYSXX7IOC7XAK27ZAXYF2ESJWEKBPBDTWG3SIQLA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.445986800 (410h3m12.317069835s) [noise=true , address=145.79.227.40:4049 , piece_id=OKMLVX2ZY2PU3O7LTIXYPICW2RN7523FAVVBKWFUYWE27MDVXPQQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.449650916 (410h3m12.320733951s) [noise=true , address=145.79.227.14:4840 , piece_id=BROYFHVRS2KNEY7NFMZQZEQXFBYR3INKWL6MS3H3TUJFRIC45TUA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.492806111 (410h3m12.363889156s) [noise=true , address=145.79.227.8:4329 , piece_id=T2Y7JZ5NIJ5ZO44CU2V5OKV5N6EGTZDD3WQPPPDA4IY3HHU2C7AQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.580529511 (410h3m12.451612546s) [noise=true , address=145.79.227.50:4375 , piece_id=N46X46ILGSTVMK4GMDAGK2S34HLQRWH66CVC3IGU5QYRXFZ3A3EQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.580551634 (410h3m12.451634679s) [seg_pos={0 0}, upload=359990]: Segment reached optimal threshold of 13 pieces. Dec 22 17:32:12.580790058 (410h3m12.451873093s) [noise=true , address=145.79.227.51:4409 , piece_id=MFKLLUWY3ZRV3WUCE6F4V54KIWA23NMQV6A47KMBNMCXY3LF6ASQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 22 17:32:12.580796850 (410h3m12.451879886s) [noise=true , address=145.79.227.51:4409 , piece_id=MFKLLUWY3ZRV3WUCE6F4V54KIWA23NMQV6A47KMBNMCXY3LF6ASQ, seg_pos={0 0}, upload=359990]: Long tail cancellation due to optimal threshold Dec 22 17:32:12.581299036 (410h3m12.452382081s) [noise=true , address=145.79.227.48:4302 , piece_id=USKG3XLIJX55W4UPB5KDGQZAFIBHKKTLVOSICOJSGTZUFIZAX7HA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF Dec 22 17:32:12.581306510 (410h3m12.452389545s) [noise=true , address=145.79.227.48:4302 , piece_id=USKG3XLIJX55W4UPB5KDGQZAFIBHKKTLVOSICOJSGTZUFIZAX7HA, seg_pos={0 0}, upload=359990]: Long tail cancellation due to optimal threshold Dec 22 17:32:12.581443571 (410h3m12.452526616s) [noise=true , address=145.79.227.36:4518 , piece_id=23OQCYXXQSLHABSRBXZSWBPCT4DEMU5QPR3C6H34BIZBKI6ZODMQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: write tcp 188.166.225.170:48704->145.79.227.36:4518: use of closed network connection; write tcp 188.166.225.170:48704->145.79.227.36:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48704->145.79.227.36:4518: use of closed network connection Dec 22 17:32:12.581450335 (410h3m12.45253338s) [noise=true , address=145.79.227.36:4518 , piece_id=23OQCYXXQSLHABSRBXZSWBPCT4DEMU5QPR3C6H34BIZBKI6ZODMQ, seg_pos={0 0}, upload=359990]: Long tail cancellation due to optimal threshold Dec 22 17:32:12.581480001 (410h3m12.452563036s) [seg_pos={0 0}, upload=359990]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:32:12.581559964 (410h3m12.452642999s) [upload=359990]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:32:12.581564973 (410h3m12.452648018s) [upload=359990]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:32:12.884501727 (410h3m12.755584762s) [upload=359990]: Done uploading segments... Dec 22 17:32:12.884522916 (410h3m12.755605951s) [upload=359990]: Upload finished. err: <nil> Duration: 7.933527049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.X.X.X...X...X.....X.........X...........X..X..XX....X...X] duration: 30.002684072s trace: https://jaeger.dev.storj.io/trace/38082a6ba78f52b1 FAIL (2025-12-22 17:32:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OOFZZGZXRX6SOWMORGLUGK7ZIOYW5Q72PPPFYYYISF3VNP3EYXMQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: U25ZF4NLSEI4Z3PCYOZRODCJ7NPRYMZF4GDER7JK4PXB7ZDWJ4BQ) context deadline exceeded ======LOGS====== Dec 22 17:32:09.352567416 (410h3m9.223650451s) [upload=359990]: Starting upload Dec 22 17:32:09.352634664 (410h3m9.223717709s) [upload=359990]: Done starting upload Dec 22 17:32:09.352830957 (410h3m9.223914002s) [upload=359990]: Uploading segments... Dec 22 17:32:09.352944304 (410h3m9.224027349s) [upload=359990]: Got next segment. Inline: false Dec 22 17:32:09.353007814 (410h3m9.224090849s) [upload=359990]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:32:09.353015149 (410h3m9.224098194s) [upload=359990]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:32:09.603456330 (410h3m9.474539375s) [seg_pos={0 0}, upload=359990]: Begin upload segment... Dec 22 17:32:09.603526253 (410h3m9.474609288s) [seg_pos={0 0}, upload=359990]: Done begin upload segment. Dec 22 17:32:09.603575257 (410h3m9.474658292s) [noise=true , address=45.91.116.6:4976 , piece_id=U25ZF4NLSEI4Z3PCYOZRODCJ7NPRYMZF4GDER7JK4PXB7ZDWJ4BQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.603620963 (410h3m9.474704008s) [noise=true , address=145.79.227.35:4481 , piece_id=TPCJMRPQW6B4WDNTEUEBXMJNE3KPM42QGGKPTXWQDVS3EDKMLHJQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.604461373 (410h3m9.475544418s) [noise=true , address=77.90.31.14:4662 , piece_id=QI53J3SAIRUBRQN6L3USYVRT5XGFYJSEOHRQG6F5R4QXKI7KRNHA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.604526648 (410h3m9.475609693s) [noise=true , address=45.91.116.9:4263 , piece_id=5XMACGPWU6SA32MJEDJ2Z2MAXKTX4HUUA7FKRTBE5YUVAAYF42PA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.604858469 (410h3m9.475941504s) [noise=true , address=145.79.227.41:4087 , piece_id=JKP6C37DWT2DZBKHFF7AMMCUABQCG6ILKNHPQU5ZRQPJB6GH4AFA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.605398318 (410h3m9.476481363s) [noise=true , address=77.90.31.11:4555 , piece_id=ZBCJ3GGEXRV3CXLN7MJNPZH4PNVQB2FMINDJRQQDGZ5RDE3WQQ5A, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.605499039 (410h3m9.476582085s) [noise=true , address=145.79.227.36:4518 , piece_id=23OQCYXXQSLHABSRBXZSWBPCT4DEMU5QPR3C6H34BIZBKI6ZODMQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.605943366 (410h3m9.477026401s) [noise=true , address=145.79.227.40:4049 , piece_id=OKMLVX2ZY2PU3O7LTIXYPICW2RN7523FAVVBKWFUYWE27MDVXPQQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.606168995 (410h3m9.47725204s) [noise=true , address=45.91.116.4:4937 , piece_id=H34DHJALOSHZENQFZBRBG7V27QZISAJBBGRRNUPF4B32LJQNOUBA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.606509494 (410h3m9.477592539s) [noise=true , address=145.79.227.8:4329 , piece_id=T2Y7JZ5NIJ5ZO44CU2V5OKV5N6EGTZDD3WQPPPDA4IY3HHU2C7AQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.606801239 (410h3m9.477884274s) [noise=true , address=145.79.227.48:4302 , piece_id=USKG3XLIJX55W4UPB5KDGQZAFIBHKKTLVOSICOJSGTZUFIZAX7HA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.606816128 (410h3m9.477899173s) [noise=true , address=145.79.227.32:4834 , piece_id=ITRX6QU4H2MQBYSXX7IOC7XAK27ZAXYF2ESJWEKBPBDTWG3SIQLA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.607357728 (410h3m9.478440773s) [noise=true , address=145.79.227.51:4409 , piece_id=MFKLLUWY3ZRV3WUCE6F4V54KIWA23NMQV6A47KMBNMCXY3LF6ASQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.607686936 (410h3m9.478769971s) [noise=true , address=45.91.116.5:4297 , piece_id=OOFZZGZXRX6SOWMORGLUGK7ZIOYW5Q72PPPFYYYISF3VNP3EYXMQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.607942712 (410h3m9.479025748s) [noise=true , address=145.79.227.50:4375 , piece_id=N46X46ILGSTVMK4GMDAGK2S34HLQRWH66CVC3IGU5QYRXFZ3A3EQ, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:09.608075656 (410h3m9.479158721s) [noise=true , address=145.79.227.14:4840 , piece_id=BROYFHVRS2KNEY7NFMZQZEQXFBYR3INKWL6MS3H3TUJFRIC45TUA, seg_pos={0 0}, upload=359990]: Uploading piece... Dec 22 17:32:11.452748346 (410h3m11.323831381s) [upload=359990]: Next returned nil segment Dec 22 17:32:11.452761570 (410h3m11.323844615s) [upload=359990]: Waiting for error group managing segments... Dec 22 17:32:11.874892199 (410h3m11.745975244s) [noise=true , address=45.91.116.4:4937 , piece_id=H34DHJALOSHZENQFZBRBG7V27QZISAJBBGRRNUPF4B32LJQNOUBA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:11.904464128 (410h3m11.775547163s) [noise=true , address=45.91.116.6:4976 , piece_id=U25ZF4NLSEI4Z3PCYOZRODCJ7NPRYMZF4GDER7JK4PXB7ZDWJ4BQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:11.994458362 (410h3m11.865541397s) [noise=true , address=45.91.116.9:4263 , piece_id=5XMACGPWU6SA32MJEDJ2Z2MAXKTX4HUUA7FKRTBE5YUVAAYF42PA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.018110329 (410h3m11.889193364s) [noise=true , address=77.90.31.11:4555 , piece_id=ZBCJ3GGEXRV3CXLN7MJNPZH4PNVQB2FMINDJRQQDGZ5RDE3WQQ5A, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.064672272 (410h3m11.935755317s) [noise=true , address=77.90.31.14:4662 , piece_id=QI53J3SAIRUBRQN6L3USYVRT5XGFYJSEOHRQG6F5R4QXKI7KRNHA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.260653673 (410h3m12.131736718s) [noise=true , address=45.91.116.5:4297 , piece_id=OOFZZGZXRX6SOWMORGLUGK7ZIOYW5Q72PPPFYYYISF3VNP3EYXMQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.380898330 (410h3m12.251981365s) [noise=true , address=145.79.227.41:4087 , piece_id=JKP6C37DWT2DZBKHFF7AMMCUABQCG6ILKNHPQU5ZRQPJB6GH4AFA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.384214434 (410h3m12.255297469s) [noise=true , address=145.79.227.35:4481 , piece_id=TPCJMRPQW6B4WDNTEUEBXMJNE3KPM42QGGKPTXWQDVS3EDKMLHJQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.396183215 (410h3m12.26726626s) [noise=true , address=145.79.227.32:4834 , piece_id=ITRX6QU4H2MQBYSXX7IOC7XAK27ZAXYF2ESJWEKBPBDTWG3SIQLA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.445986800 (410h3m12.317069835s) [noise=true , address=145.79.227.40:4049 , piece_id=OKMLVX2ZY2PU3O7LTIXYPICW2RN7523FAVVBKWFUYWE27MDVXPQQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.449650916 (410h3m12.320733951s) [noise=true , address=145.79.227.14:4840 , piece_id=BROYFHVRS2KNEY7NFMZQZEQXFBYR3INKWL6MS3H3TUJFRIC45TUA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.492806111 (410h3m12.363889156s) [noise=true , address=145.79.227.8:4329 , piece_id=T2Y7JZ5NIJ5ZO44CU2V5OKV5N6EGTZDD3WQPPPDA4IY3HHU2C7AQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.580529511 (410h3m12.451612546s) [noise=true , address=145.79.227.50:4375 , piece_id=N46X46ILGSTVMK4GMDAGK2S34HLQRWH66CVC3IGU5QYRXFZ3A3EQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: <nil> Dec 22 17:32:12.580551634 (410h3m12.451634679s) [seg_pos={0 0}, upload=359990]: Segment reached optimal threshold of 13 pieces. Dec 22 17:32:12.580790058 (410h3m12.451873093s) [noise=true , address=145.79.227.51:4409 , piece_id=MFKLLUWY3ZRV3WUCE6F4V54KIWA23NMQV6A47KMBNMCXY3LF6ASQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 22 17:32:12.580796850 (410h3m12.451879886s) [noise=true , address=145.79.227.51:4409 , piece_id=MFKLLUWY3ZRV3WUCE6F4V54KIWA23NMQV6A47KMBNMCXY3LF6ASQ, seg_pos={0 0}, upload=359990]: Long tail cancellation due to optimal threshold Dec 22 17:32:12.581299036 (410h3m12.452382081s) [noise=true , address=145.79.227.48:4302 , piece_id=USKG3XLIJX55W4UPB5KDGQZAFIBHKKTLVOSICOJSGTZUFIZAX7HA, seg_pos={0 0}, upload=359990]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF Dec 22 17:32:12.581306510 (410h3m12.452389545s) [noise=true , address=145.79.227.48:4302 , piece_id=USKG3XLIJX55W4UPB5KDGQZAFIBHKKTLVOSICOJSGTZUFIZAX7HA, seg_pos={0 0}, upload=359990]: Long tail cancellation due to optimal threshold Dec 22 17:32:12.581443571 (410h3m12.452526616s) [noise=true , address=145.79.227.36:4518 , piece_id=23OQCYXXQSLHABSRBXZSWBPCT4DEMU5QPR3C6H34BIZBKI6ZODMQ, seg_pos={0 0}, upload=359990]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: write tcp 188.166.225.170:48704->145.79.227.36:4518: use of closed network connection; write tcp 188.166.225.170:48704->145.79.227.36:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48704->145.79.227.36:4518: use of closed network connection Dec 22 17:32:12.581450335 (410h3m12.45253338s) [noise=true , address=145.79.227.36:4518 , piece_id=23OQCYXXQSLHABSRBXZSWBPCT4DEMU5QPR3C6H34BIZBKI6ZODMQ, seg_pos={0 0}, upload=359990]: Long tail cancellation due to optimal threshold Dec 22 17:32:12.581480001 (410h3m12.452563036s) [seg_pos={0 0}, upload=359990]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:32:12.581559964 (410h3m12.452642999s) [upload=359990]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:32:12.581564973 (410h3m12.452648018s) [upload=359990]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:32:12.884501727 (410h3m12.755584762s) [upload=359990]: Done uploading segments... Dec 22 17:32:12.884522916 (410h3m12.755605951s) [upload=359990]: Upload finished. err: <nil> Duration: 30.002706995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X.X.X.X...X...X.....X.........X...........X..X..XX....X...] duration: 23.096174784s trace: https://jaeger.dev.storj.io/trace/1725d06c37d78428 FAIL (2025-12-22 17:31:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A) context deadline exceeded ======LOGS====== Dec 22 17:28:11.556010126 (409h59m11.427093161s) [upload=359931]: Starting upload Dec 22 17:28:11.556070852 (409h59m11.427153887s) [upload=359931]: Done starting upload Dec 22 17:28:11.556242318 (409h59m11.427325363s) [upload=359931]: Uploading segments... Dec 22 17:28:11.556373267 (409h59m11.427456302s) [upload=359931]: Got next segment. Inline: false Dec 22 17:28:11.556472436 (409h59m11.427555471s) [upload=359931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:28:11.556508955 (409h59m11.427592s) [upload=359931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:28:11.799228633 (409h59m11.670311668s) [seg_pos={0 0}, upload=359931]: Begin upload segment... Dec 22 17:28:11.799276784 (409h59m11.670359819s) [seg_pos={0 0}, upload=359931]: Done begin upload segment. Dec 22 17:28:11.799300921 (409h59m11.670383956s) [noise=true , address=145.79.227.17:4972 , piece_id=MX5BVOQ24IYEOPHHH4SAKLQPIBYL7DMD3BWPDPQG6G5HDDOUTINA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.799405510 (409h59m11.670488545s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800125149 (409h59m11.671208194s) [noise=true , address=145.79.227.11:4732 , piece_id=EB4LBADEPJ3TJO5WNQYLUQNHSBUCMLT5TEW7IRWFOSMLUPR4EHSA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800347604 (409h59m11.671430639s) [noise=true , address=145.79.227.29:4941 , piece_id=XV7WBEI57VJEJLK43EDMFBVQRU3INIA7FA43F36YFHZJ7KC5FVTA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800704102 (409h59m11.671787137s) [noise=true , address=45.91.116.12:4226 , piece_id=E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801266092 (409h59m11.672349137s) [noise=true , address=145.79.227.32:4835 , piece_id=SNA6MQWEDCEBTWOODLMQ5YN5I7FRXACEZJNX25ZCOZOY3EGKV4QQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801311750 (409h59m11.672394795s) [noise=true , address=145.79.227.47:4266 , piece_id=YQW7O3J63MAMFWNZW34GF7ZWUVTGNCFBONXYSG3AF6S2QGLSNSFQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801978169 (409h59m11.673061204s) [noise=true , address=77.90.31.17:4770 , piece_id=XBXCG3MQON3TDOI44L7JGMWCQFFVOCDQOHMKDCWL4L6OZH43ZFFA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.802578501 (409h59m11.673661537s) [noise=true , address=145.79.227.39:4013 , piece_id=MKBGPPQ2XL65Q65X4ETXTFCAIOX4M75DA5VUZH53MJ4IYQZ2UOLA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803060009 (409h59m11.674143054s) [noise=true , address=45.91.116.8:4047 , piece_id=WTJ57FNXW7RDAHU4EMICW6257DKHI7D3EXY746DFAEGQKDZ4EKTA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803076089 (409h59m11.674159124s) [noise=true , address=145.79.227.35:4483 , piece_id=TRMDO2AYRS5YTJ4KNXTSYU6PM5NI7XGE3ZE4KYPKLQB6CWR7MAPQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803421997 (409h59m11.674505032s) [noise=true , address=45.91.116.9:4261 , piece_id=HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803950464 (409h59m11.675033499s) [noise=true , address=45.91.116.7:4010 , piece_id=L5NOCYA6XOUFQJHIKBXQMZ6TGXO3FVXTYM6SANTPAD4LLAMMFBBA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.804700021 (409h59m11.675783066s) [noise=true , address=145.79.227.12:4765 , piece_id=ZYJGAITY4WXGNXORE6XNJQYNKC6NA2MJOZL6DU76QJAVBC6SZONA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.806281182 (409h59m11.677364217s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.808423962 (409h59m11.679506997s) [noise=true , address=145.79.227.42:4121 , piece_id=ZXUB76EK6HQMUM2GFHMITMSW6DXCEOE7SXQP743W6V2AACH57AHQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:12.960526870 (409h59m12.831609905s) [noise=true , address=45.91.116.8:4047 , piece_id=WTJ57FNXW7RDAHU4EMICW6257DKHI7D3EXY746DFAEGQKDZ4EKTA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: storage node overloaded, request limit: 2000 Dec 22 17:28:13.582863800 (409h59m13.453946835s) [upload=359931]: Next returned nil segment Dec 22 17:28:13.582875993 (409h59m13.453959028s) [upload=359931]: Waiting for error group managing segments... Dec 22 17:28:14.007762556 (409h59m13.878845591s) [noise=true , address=45.91.116.12:4226 , piece_id=E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.085398697 (409h59m13.956481742s) [noise=true , address=77.90.31.17:4770 , piece_id=XBXCG3MQON3TDOI44L7JGMWCQFFVOCDQOHMKDCWL4L6OZH43ZFFA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.121342951 (409h59m13.992425996s) [noise=true , address=45.91.116.9:4261 , piece_id=HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.650298637 (409h59m14.521381673s) [noise=true , address=145.79.227.35:4483 , piece_id=TRMDO2AYRS5YTJ4KNXTSYU6PM5NI7XGE3ZE4KYPKLQB6CWR7MAPQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.676099536 (409h59m14.547182581s) [noise=true , address=145.79.227.32:4835 , piece_id=SNA6MQWEDCEBTWOODLMQ5YN5I7FRXACEZJNX25ZCOZOY3EGKV4QQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.680791089 (409h59m14.551874124s) [noise=true , address=145.79.227.11:4732 , piece_id=EB4LBADEPJ3TJO5WNQYLUQNHSBUCMLT5TEW7IRWFOSMLUPR4EHSA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.715950850 (409h59m14.587033885s) [noise=true , address=145.79.227.47:4266 , piece_id=YQW7O3J63MAMFWNZW34GF7ZWUVTGNCFBONXYSG3AF6S2QGLSNSFQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.718896127 (409h59m14.589979162s) [noise=true , address=145.79.227.39:4013 , piece_id=MKBGPPQ2XL65Q65X4ETXTFCAIOX4M75DA5VUZH53MJ4IYQZ2UOLA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.719923644 (409h59m14.591006679s) [noise=true , address=145.79.227.12:4765 , piece_id=ZYJGAITY4WXGNXORE6XNJQYNKC6NA2MJOZL6DU76QJAVBC6SZONA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.727259453 (409h59m14.598342488s) [noise=true , address=145.79.227.17:4972 , piece_id=MX5BVOQ24IYEOPHHH4SAKLQPIBYL7DMD3BWPDPQG6G5HDDOUTINA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.738092731 (409h59m14.609175766s) [noise=true , address=145.79.227.42:4121 , piece_id=ZXUB76EK6HQMUM2GFHMITMSW6DXCEOE7SXQP743W6V2AACH57AHQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.740445641 (409h59m14.611528676s) [noise=true , address=145.79.227.29:4941 , piece_id=XV7WBEI57VJEJLK43EDMFBVQRU3INIA7FA43F36YFHZJ7KC5FVTA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.790262851 (409h59m14.661345886s) [noise=true , address=45.91.116.7:4010 , piece_id=L5NOCYA6XOUFQJHIKBXQMZ6TGXO3FVXTYM6SANTPAD4LLAMMFBBA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.790283420 (409h59m14.661366455s) [seg_pos={0 0}, upload=359931]: Segment reached optimal threshold of 13 pieces. Dec 22 17:28:14.790522496 (409h59m14.661605531s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 22 17:28:14.790529970 (409h59m14.661613005s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Long tail cancellation due to optimal threshold Dec 22 17:28:14.790690486 (409h59m14.661773521s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 22 17:28:14.790705885 (409h59m14.66178893s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Long tail cancellation due to optimal threshold Dec 22 17:28:14.790728759 (409h59m14.661811804s) [seg_pos={0 0}, upload=359931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:28:14.790815834 (409h59m14.661898869s) [upload=359931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:28:14.790822336 (409h59m14.661905371s) [upload=359931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:28:15.083908209 (409h59m14.954991254s) [upload=359931]: Done uploading segments... Dec 22 17:28:15.083925302 (409h59m14.955008347s) [upload=359931]: Upload finished. err: <nil> Duration: 23.096201033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX.X.X.X.X...X...X.....X.........X...........X..X..XX....X..] duration: 25.876892227s trace: https://jaeger.dev.storj.io/trace/1725d06c37d78428 FAIL (2025-12-22 17:30:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A) context deadline exceeded ======LOGS====== Dec 22 17:28:11.556010126 (409h59m11.427093161s) [upload=359931]: Starting upload Dec 22 17:28:11.556070852 (409h59m11.427153887s) [upload=359931]: Done starting upload Dec 22 17:28:11.556242318 (409h59m11.427325363s) [upload=359931]: Uploading segments... Dec 22 17:28:11.556373267 (409h59m11.427456302s) [upload=359931]: Got next segment. Inline: false Dec 22 17:28:11.556472436 (409h59m11.427555471s) [upload=359931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:28:11.556508955 (409h59m11.427592s) [upload=359931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:28:11.799228633 (409h59m11.670311668s) [seg_pos={0 0}, upload=359931]: Begin upload segment... Dec 22 17:28:11.799276784 (409h59m11.670359819s) [seg_pos={0 0}, upload=359931]: Done begin upload segment. Dec 22 17:28:11.799300921 (409h59m11.670383956s) [noise=true , address=145.79.227.17:4972 , piece_id=MX5BVOQ24IYEOPHHH4SAKLQPIBYL7DMD3BWPDPQG6G5HDDOUTINA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.799405510 (409h59m11.670488545s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800125149 (409h59m11.671208194s) [noise=true , address=145.79.227.11:4732 , piece_id=EB4LBADEPJ3TJO5WNQYLUQNHSBUCMLT5TEW7IRWFOSMLUPR4EHSA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800347604 (409h59m11.671430639s) [noise=true , address=145.79.227.29:4941 , piece_id=XV7WBEI57VJEJLK43EDMFBVQRU3INIA7FA43F36YFHZJ7KC5FVTA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800704102 (409h59m11.671787137s) [noise=true , address=45.91.116.12:4226 , piece_id=E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801266092 (409h59m11.672349137s) [noise=true , address=145.79.227.32:4835 , piece_id=SNA6MQWEDCEBTWOODLMQ5YN5I7FRXACEZJNX25ZCOZOY3EGKV4QQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801311750 (409h59m11.672394795s) [noise=true , address=145.79.227.47:4266 , piece_id=YQW7O3J63MAMFWNZW34GF7ZWUVTGNCFBONXYSG3AF6S2QGLSNSFQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801978169 (409h59m11.673061204s) [noise=true , address=77.90.31.17:4770 , piece_id=XBXCG3MQON3TDOI44L7JGMWCQFFVOCDQOHMKDCWL4L6OZH43ZFFA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.802578501 (409h59m11.673661537s) [noise=true , address=145.79.227.39:4013 , piece_id=MKBGPPQ2XL65Q65X4ETXTFCAIOX4M75DA5VUZH53MJ4IYQZ2UOLA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803060009 (409h59m11.674143054s) [noise=true , address=45.91.116.8:4047 , piece_id=WTJ57FNXW7RDAHU4EMICW6257DKHI7D3EXY746DFAEGQKDZ4EKTA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803076089 (409h59m11.674159124s) [noise=true , address=145.79.227.35:4483 , piece_id=TRMDO2AYRS5YTJ4KNXTSYU6PM5NI7XGE3ZE4KYPKLQB6CWR7MAPQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803421997 (409h59m11.674505032s) [noise=true , address=45.91.116.9:4261 , piece_id=HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803950464 (409h59m11.675033499s) [noise=true , address=45.91.116.7:4010 , piece_id=L5NOCYA6XOUFQJHIKBXQMZ6TGXO3FVXTYM6SANTPAD4LLAMMFBBA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.804700021 (409h59m11.675783066s) [noise=true , address=145.79.227.12:4765 , piece_id=ZYJGAITY4WXGNXORE6XNJQYNKC6NA2MJOZL6DU76QJAVBC6SZONA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.806281182 (409h59m11.677364217s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.808423962 (409h59m11.679506997s) [noise=true , address=145.79.227.42:4121 , piece_id=ZXUB76EK6HQMUM2GFHMITMSW6DXCEOE7SXQP743W6V2AACH57AHQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:12.960526870 (409h59m12.831609905s) [noise=true , address=45.91.116.8:4047 , piece_id=WTJ57FNXW7RDAHU4EMICW6257DKHI7D3EXY746DFAEGQKDZ4EKTA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: storage node overloaded, request limit: 2000 Dec 22 17:28:13.582863800 (409h59m13.453946835s) [upload=359931]: Next returned nil segment Dec 22 17:28:13.582875993 (409h59m13.453959028s) [upload=359931]: Waiting for error group managing segments... Dec 22 17:28:14.007762556 (409h59m13.878845591s) [noise=true , address=45.91.116.12:4226 , piece_id=E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.085398697 (409h59m13.956481742s) [noise=true , address=77.90.31.17:4770 , piece_id=XBXCG3MQON3TDOI44L7JGMWCQFFVOCDQOHMKDCWL4L6OZH43ZFFA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.121342951 (409h59m13.992425996s) [noise=true , address=45.91.116.9:4261 , piece_id=HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.650298637 (409h59m14.521381673s) [noise=true , address=145.79.227.35:4483 , piece_id=TRMDO2AYRS5YTJ4KNXTSYU6PM5NI7XGE3ZE4KYPKLQB6CWR7MAPQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.676099536 (409h59m14.547182581s) [noise=true , address=145.79.227.32:4835 , piece_id=SNA6MQWEDCEBTWOODLMQ5YN5I7FRXACEZJNX25ZCOZOY3EGKV4QQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.680791089 (409h59m14.551874124s) [noise=true , address=145.79.227.11:4732 , piece_id=EB4LBADEPJ3TJO5WNQYLUQNHSBUCMLT5TEW7IRWFOSMLUPR4EHSA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.715950850 (409h59m14.587033885s) [noise=true , address=145.79.227.47:4266 , piece_id=YQW7O3J63MAMFWNZW34GF7ZWUVTGNCFBONXYSG3AF6S2QGLSNSFQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.718896127 (409h59m14.589979162s) [noise=true , address=145.79.227.39:4013 , piece_id=MKBGPPQ2XL65Q65X4ETXTFCAIOX4M75DA5VUZH53MJ4IYQZ2UOLA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.719923644 (409h59m14.591006679s) [noise=true , address=145.79.227.12:4765 , piece_id=ZYJGAITY4WXGNXORE6XNJQYNKC6NA2MJOZL6DU76QJAVBC6SZONA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.727259453 (409h59m14.598342488s) [noise=true , address=145.79.227.17:4972 , piece_id=MX5BVOQ24IYEOPHHH4SAKLQPIBYL7DMD3BWPDPQG6G5HDDOUTINA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.738092731 (409h59m14.609175766s) [noise=true , address=145.79.227.42:4121 , piece_id=ZXUB76EK6HQMUM2GFHMITMSW6DXCEOE7SXQP743W6V2AACH57AHQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.740445641 (409h59m14.611528676s) [noise=true , address=145.79.227.29:4941 , piece_id=XV7WBEI57VJEJLK43EDMFBVQRU3INIA7FA43F36YFHZJ7KC5FVTA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.790262851 (409h59m14.661345886s) [noise=true , address=45.91.116.7:4010 , piece_id=L5NOCYA6XOUFQJHIKBXQMZ6TGXO3FVXTYM6SANTPAD4LLAMMFBBA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.790283420 (409h59m14.661366455s) [seg_pos={0 0}, upload=359931]: Segment reached optimal threshold of 13 pieces. Dec 22 17:28:14.790522496 (409h59m14.661605531s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 22 17:28:14.790529970 (409h59m14.661613005s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Long tail cancellation due to optimal threshold Dec 22 17:28:14.790690486 (409h59m14.661773521s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 22 17:28:14.790705885 (409h59m14.66178893s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Long tail cancellation due to optimal threshold Dec 22 17:28:14.790728759 (409h59m14.661811804s) [seg_pos={0 0}, upload=359931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:28:14.790815834 (409h59m14.661898869s) [upload=359931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:28:14.790822336 (409h59m14.661905371s) [upload=359931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:28:15.083908209 (409h59m14.954991254s) [upload=359931]: Done uploading segments... Dec 22 17:28:15.083925302 (409h59m14.955008347s) [upload=359931]: Upload finished. err: <nil> Duration: 25.876921494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX.X.X.X.X...X...X.....X.........X...........X..X..XX....X.] duration: 27.210176289s trace: https://jaeger.dev.storj.io/trace/1725d06c37d78428 FAIL (2025-12-22 17:29:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A) context deadline exceeded ======LOGS====== Dec 22 17:28:11.556010126 (409h59m11.427093161s) [upload=359931]: Starting upload Dec 22 17:28:11.556070852 (409h59m11.427153887s) [upload=359931]: Done starting upload Dec 22 17:28:11.556242318 (409h59m11.427325363s) [upload=359931]: Uploading segments... Dec 22 17:28:11.556373267 (409h59m11.427456302s) [upload=359931]: Got next segment. Inline: false Dec 22 17:28:11.556472436 (409h59m11.427555471s) [upload=359931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:28:11.556508955 (409h59m11.427592s) [upload=359931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:28:11.799228633 (409h59m11.670311668s) [seg_pos={0 0}, upload=359931]: Begin upload segment... Dec 22 17:28:11.799276784 (409h59m11.670359819s) [seg_pos={0 0}, upload=359931]: Done begin upload segment. Dec 22 17:28:11.799300921 (409h59m11.670383956s) [noise=true , address=145.79.227.17:4972 , piece_id=MX5BVOQ24IYEOPHHH4SAKLQPIBYL7DMD3BWPDPQG6G5HDDOUTINA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.799405510 (409h59m11.670488545s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800125149 (409h59m11.671208194s) [noise=true , address=145.79.227.11:4732 , piece_id=EB4LBADEPJ3TJO5WNQYLUQNHSBUCMLT5TEW7IRWFOSMLUPR4EHSA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800347604 (409h59m11.671430639s) [noise=true , address=145.79.227.29:4941 , piece_id=XV7WBEI57VJEJLK43EDMFBVQRU3INIA7FA43F36YFHZJ7KC5FVTA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800704102 (409h59m11.671787137s) [noise=true , address=45.91.116.12:4226 , piece_id=E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801266092 (409h59m11.672349137s) [noise=true , address=145.79.227.32:4835 , piece_id=SNA6MQWEDCEBTWOODLMQ5YN5I7FRXACEZJNX25ZCOZOY3EGKV4QQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801311750 (409h59m11.672394795s) [noise=true , address=145.79.227.47:4266 , piece_id=YQW7O3J63MAMFWNZW34GF7ZWUVTGNCFBONXYSG3AF6S2QGLSNSFQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801978169 (409h59m11.673061204s) [noise=true , address=77.90.31.17:4770 , piece_id=XBXCG3MQON3TDOI44L7JGMWCQFFVOCDQOHMKDCWL4L6OZH43ZFFA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.802578501 (409h59m11.673661537s) [noise=true , address=145.79.227.39:4013 , piece_id=MKBGPPQ2XL65Q65X4ETXTFCAIOX4M75DA5VUZH53MJ4IYQZ2UOLA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803060009 (409h59m11.674143054s) [noise=true , address=45.91.116.8:4047 , piece_id=WTJ57FNXW7RDAHU4EMICW6257DKHI7D3EXY746DFAEGQKDZ4EKTA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803076089 (409h59m11.674159124s) [noise=true , address=145.79.227.35:4483 , piece_id=TRMDO2AYRS5YTJ4KNXTSYU6PM5NI7XGE3ZE4KYPKLQB6CWR7MAPQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803421997 (409h59m11.674505032s) [noise=true , address=45.91.116.9:4261 , piece_id=HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803950464 (409h59m11.675033499s) [noise=true , address=45.91.116.7:4010 , piece_id=L5NOCYA6XOUFQJHIKBXQMZ6TGXO3FVXTYM6SANTPAD4LLAMMFBBA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.804700021 (409h59m11.675783066s) [noise=true , address=145.79.227.12:4765 , piece_id=ZYJGAITY4WXGNXORE6XNJQYNKC6NA2MJOZL6DU76QJAVBC6SZONA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.806281182 (409h59m11.677364217s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.808423962 (409h59m11.679506997s) [noise=true , address=145.79.227.42:4121 , piece_id=ZXUB76EK6HQMUM2GFHMITMSW6DXCEOE7SXQP743W6V2AACH57AHQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:12.960526870 (409h59m12.831609905s) [noise=true , address=45.91.116.8:4047 , piece_id=WTJ57FNXW7RDAHU4EMICW6257DKHI7D3EXY746DFAEGQKDZ4EKTA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: storage node overloaded, request limit: 2000 Dec 22 17:28:13.582863800 (409h59m13.453946835s) [upload=359931]: Next returned nil segment Dec 22 17:28:13.582875993 (409h59m13.453959028s) [upload=359931]: Waiting for error group managing segments... Dec 22 17:28:14.007762556 (409h59m13.878845591s) [noise=true , address=45.91.116.12:4226 , piece_id=E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.085398697 (409h59m13.956481742s) [noise=true , address=77.90.31.17:4770 , piece_id=XBXCG3MQON3TDOI44L7JGMWCQFFVOCDQOHMKDCWL4L6OZH43ZFFA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.121342951 (409h59m13.992425996s) [noise=true , address=45.91.116.9:4261 , piece_id=HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.650298637 (409h59m14.521381673s) [noise=true , address=145.79.227.35:4483 , piece_id=TRMDO2AYRS5YTJ4KNXTSYU6PM5NI7XGE3ZE4KYPKLQB6CWR7MAPQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.676099536 (409h59m14.547182581s) [noise=true , address=145.79.227.32:4835 , piece_id=SNA6MQWEDCEBTWOODLMQ5YN5I7FRXACEZJNX25ZCOZOY3EGKV4QQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.680791089 (409h59m14.551874124s) [noise=true , address=145.79.227.11:4732 , piece_id=EB4LBADEPJ3TJO5WNQYLUQNHSBUCMLT5TEW7IRWFOSMLUPR4EHSA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.715950850 (409h59m14.587033885s) [noise=true , address=145.79.227.47:4266 , piece_id=YQW7O3J63MAMFWNZW34GF7ZWUVTGNCFBONXYSG3AF6S2QGLSNSFQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.718896127 (409h59m14.589979162s) [noise=true , address=145.79.227.39:4013 , piece_id=MKBGPPQ2XL65Q65X4ETXTFCAIOX4M75DA5VUZH53MJ4IYQZ2UOLA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.719923644 (409h59m14.591006679s) [noise=true , address=145.79.227.12:4765 , piece_id=ZYJGAITY4WXGNXORE6XNJQYNKC6NA2MJOZL6DU76QJAVBC6SZONA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.727259453 (409h59m14.598342488s) [noise=true , address=145.79.227.17:4972 , piece_id=MX5BVOQ24IYEOPHHH4SAKLQPIBYL7DMD3BWPDPQG6G5HDDOUTINA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.738092731 (409h59m14.609175766s) [noise=true , address=145.79.227.42:4121 , piece_id=ZXUB76EK6HQMUM2GFHMITMSW6DXCEOE7SXQP743W6V2AACH57AHQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.740445641 (409h59m14.611528676s) [noise=true , address=145.79.227.29:4941 , piece_id=XV7WBEI57VJEJLK43EDMFBVQRU3INIA7FA43F36YFHZJ7KC5FVTA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.790262851 (409h59m14.661345886s) [noise=true , address=45.91.116.7:4010 , piece_id=L5NOCYA6XOUFQJHIKBXQMZ6TGXO3FVXTYM6SANTPAD4LLAMMFBBA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.790283420 (409h59m14.661366455s) [seg_pos={0 0}, upload=359931]: Segment reached optimal threshold of 13 pieces. Dec 22 17:28:14.790522496 (409h59m14.661605531s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 22 17:28:14.790529970 (409h59m14.661613005s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Long tail cancellation due to optimal threshold Dec 22 17:28:14.790690486 (409h59m14.661773521s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 22 17:28:14.790705885 (409h59m14.66178893s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Long tail cancellation due to optimal threshold Dec 22 17:28:14.790728759 (409h59m14.661811804s) [seg_pos={0 0}, upload=359931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:28:14.790815834 (409h59m14.661898869s) [upload=359931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:28:14.790822336 (409h59m14.661905371s) [upload=359931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:28:15.083908209 (409h59m14.954991254s) [upload=359931]: Done uploading segments... Dec 22 17:28:15.083925302 (409h59m14.955008347s) [upload=359931]: Upload finished. err: <nil> Duration: 27.210206394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.XX.X.X.X.X...X...X.....X.........X...........X..X..XX....X] duration: 30.002660941s trace: https://jaeger.dev.storj.io/trace/1725d06c37d78428 FAIL (2025-12-22 17:28:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A) context deadline exceeded ======LOGS====== Dec 22 17:28:11.556010126 (409h59m11.427093161s) [upload=359931]: Starting upload Dec 22 17:28:11.556070852 (409h59m11.427153887s) [upload=359931]: Done starting upload Dec 22 17:28:11.556242318 (409h59m11.427325363s) [upload=359931]: Uploading segments... Dec 22 17:28:11.556373267 (409h59m11.427456302s) [upload=359931]: Got next segment. Inline: false Dec 22 17:28:11.556472436 (409h59m11.427555471s) [upload=359931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:28:11.556508955 (409h59m11.427592s) [upload=359931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:28:11.799228633 (409h59m11.670311668s) [seg_pos={0 0}, upload=359931]: Begin upload segment... Dec 22 17:28:11.799276784 (409h59m11.670359819s) [seg_pos={0 0}, upload=359931]: Done begin upload segment. Dec 22 17:28:11.799300921 (409h59m11.670383956s) [noise=true , address=145.79.227.17:4972 , piece_id=MX5BVOQ24IYEOPHHH4SAKLQPIBYL7DMD3BWPDPQG6G5HDDOUTINA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.799405510 (409h59m11.670488545s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800125149 (409h59m11.671208194s) [noise=true , address=145.79.227.11:4732 , piece_id=EB4LBADEPJ3TJO5WNQYLUQNHSBUCMLT5TEW7IRWFOSMLUPR4EHSA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800347604 (409h59m11.671430639s) [noise=true , address=145.79.227.29:4941 , piece_id=XV7WBEI57VJEJLK43EDMFBVQRU3INIA7FA43F36YFHZJ7KC5FVTA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.800704102 (409h59m11.671787137s) [noise=true , address=45.91.116.12:4226 , piece_id=E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801266092 (409h59m11.672349137s) [noise=true , address=145.79.227.32:4835 , piece_id=SNA6MQWEDCEBTWOODLMQ5YN5I7FRXACEZJNX25ZCOZOY3EGKV4QQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801311750 (409h59m11.672394795s) [noise=true , address=145.79.227.47:4266 , piece_id=YQW7O3J63MAMFWNZW34GF7ZWUVTGNCFBONXYSG3AF6S2QGLSNSFQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.801978169 (409h59m11.673061204s) [noise=true , address=77.90.31.17:4770 , piece_id=XBXCG3MQON3TDOI44L7JGMWCQFFVOCDQOHMKDCWL4L6OZH43ZFFA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.802578501 (409h59m11.673661537s) [noise=true , address=145.79.227.39:4013 , piece_id=MKBGPPQ2XL65Q65X4ETXTFCAIOX4M75DA5VUZH53MJ4IYQZ2UOLA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803060009 (409h59m11.674143054s) [noise=true , address=45.91.116.8:4047 , piece_id=WTJ57FNXW7RDAHU4EMICW6257DKHI7D3EXY746DFAEGQKDZ4EKTA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803076089 (409h59m11.674159124s) [noise=true , address=145.79.227.35:4483 , piece_id=TRMDO2AYRS5YTJ4KNXTSYU6PM5NI7XGE3ZE4KYPKLQB6CWR7MAPQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803421997 (409h59m11.674505032s) [noise=true , address=45.91.116.9:4261 , piece_id=HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.803950464 (409h59m11.675033499s) [noise=true , address=45.91.116.7:4010 , piece_id=L5NOCYA6XOUFQJHIKBXQMZ6TGXO3FVXTYM6SANTPAD4LLAMMFBBA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.804700021 (409h59m11.675783066s) [noise=true , address=145.79.227.12:4765 , piece_id=ZYJGAITY4WXGNXORE6XNJQYNKC6NA2MJOZL6DU76QJAVBC6SZONA, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.806281182 (409h59m11.677364217s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:11.808423962 (409h59m11.679506997s) [noise=true , address=145.79.227.42:4121 , piece_id=ZXUB76EK6HQMUM2GFHMITMSW6DXCEOE7SXQP743W6V2AACH57AHQ, seg_pos={0 0}, upload=359931]: Uploading piece... Dec 22 17:28:12.960526870 (409h59m12.831609905s) [noise=true , address=45.91.116.8:4047 , piece_id=WTJ57FNXW7RDAHU4EMICW6257DKHI7D3EXY746DFAEGQKDZ4EKTA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: storage node overloaded, request limit: 2000 Dec 22 17:28:13.582863800 (409h59m13.453946835s) [upload=359931]: Next returned nil segment Dec 22 17:28:13.582875993 (409h59m13.453959028s) [upload=359931]: Waiting for error group managing segments... Dec 22 17:28:14.007762556 (409h59m13.878845591s) [noise=true , address=45.91.116.12:4226 , piece_id=E7EZRUQMPH2LPKLFOHPNHNPSCNNRQ7QZUW6SKNS2SI7AFCSHLF7A, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.085398697 (409h59m13.956481742s) [noise=true , address=77.90.31.17:4770 , piece_id=XBXCG3MQON3TDOI44L7JGMWCQFFVOCDQOHMKDCWL4L6OZH43ZFFA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.121342951 (409h59m13.992425996s) [noise=true , address=45.91.116.9:4261 , piece_id=HWGJGCXTKBE6JCUKAK3HS5XMERGM3KZB2A7CC56C6KC6PYGJYQXQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.650298637 (409h59m14.521381673s) [noise=true , address=145.79.227.35:4483 , piece_id=TRMDO2AYRS5YTJ4KNXTSYU6PM5NI7XGE3ZE4KYPKLQB6CWR7MAPQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.676099536 (409h59m14.547182581s) [noise=true , address=145.79.227.32:4835 , piece_id=SNA6MQWEDCEBTWOODLMQ5YN5I7FRXACEZJNX25ZCOZOY3EGKV4QQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.680791089 (409h59m14.551874124s) [noise=true , address=145.79.227.11:4732 , piece_id=EB4LBADEPJ3TJO5WNQYLUQNHSBUCMLT5TEW7IRWFOSMLUPR4EHSA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.715950850 (409h59m14.587033885s) [noise=true , address=145.79.227.47:4266 , piece_id=YQW7O3J63MAMFWNZW34GF7ZWUVTGNCFBONXYSG3AF6S2QGLSNSFQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.718896127 (409h59m14.589979162s) [noise=true , address=145.79.227.39:4013 , piece_id=MKBGPPQ2XL65Q65X4ETXTFCAIOX4M75DA5VUZH53MJ4IYQZ2UOLA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.719923644 (409h59m14.591006679s) [noise=true , address=145.79.227.12:4765 , piece_id=ZYJGAITY4WXGNXORE6XNJQYNKC6NA2MJOZL6DU76QJAVBC6SZONA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.727259453 (409h59m14.598342488s) [noise=true , address=145.79.227.17:4972 , piece_id=MX5BVOQ24IYEOPHHH4SAKLQPIBYL7DMD3BWPDPQG6G5HDDOUTINA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.738092731 (409h59m14.609175766s) [noise=true , address=145.79.227.42:4121 , piece_id=ZXUB76EK6HQMUM2GFHMITMSW6DXCEOE7SXQP743W6V2AACH57AHQ, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.740445641 (409h59m14.611528676s) [noise=true , address=145.79.227.29:4941 , piece_id=XV7WBEI57VJEJLK43EDMFBVQRU3INIA7FA43F36YFHZJ7KC5FVTA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.790262851 (409h59m14.661345886s) [noise=true , address=45.91.116.7:4010 , piece_id=L5NOCYA6XOUFQJHIKBXQMZ6TGXO3FVXTYM6SANTPAD4LLAMMFBBA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: <nil> Dec 22 17:28:14.790283420 (409h59m14.661366455s) [seg_pos={0 0}, upload=359931]: Segment reached optimal threshold of 13 pieces. Dec 22 17:28:14.790522496 (409h59m14.661605531s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 22 17:28:14.790529970 (409h59m14.661613005s) [noise=true , address=145.79.227.30:4761 , piece_id=DRT3N2QQMOS57KJIUCYSABDK2KWGEOUZK64C7BFAUKNGOYX2ZSBA, seg_pos={0 0}, upload=359931]: Long tail cancellation due to optimal threshold Dec 22 17:28:14.790690486 (409h59m14.661773521s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 22 17:28:14.790705885 (409h59m14.66178893s) [noise=true , address=145.79.227.33:4870 , piece_id=L3AKNPDXNYYRWA3QDZVMKNMDHBQMWOC4EL35MNGIST73573PYL3A, seg_pos={0 0}, upload=359931]: Long tail cancellation due to optimal threshold Dec 22 17:28:14.790728759 (409h59m14.661811804s) [seg_pos={0 0}, upload=359931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:28:14.790815834 (409h59m14.661898869s) [upload=359931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:28:14.790822336 (409h59m14.661905371s) [upload=359931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:28:15.083908209 (409h59m14.954991254s) [upload=359931]: Done uploading segments... Dec 22 17:28:15.083925302 (409h59m14.955008347s) [upload=359931]: Upload finished. err: <nil> Duration: 30.002684306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.XX.X.X.X.X...X...X.....X.........X...........X..X..XX....] duration: 7.4448692s trace: https://jaeger.dev.storj.io/trace/120491d8b53108cc FAIL (2025-12-22 17:27:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 17:25:17.316020904 (409h56m17.187103949s) [upload=359886]: Starting upload Dec 22 17:25:17.316097080 (409h56m17.187180125s) [upload=359886]: Done starting upload Dec 22 17:25:17.316317088 (409h56m17.187400134s) [upload=359886]: Uploading segments... Dec 22 17:25:17.316356754 (409h56m17.187439789s) [upload=359886]: Got next segment. Inline: false Dec 22 17:25:17.316384707 (409h56m17.187467752s) [upload=359886]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:25:17.316390418 (409h56m17.187473453s) [upload=359886]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:25:17.560587348 (409h56m17.431670393s) [seg_pos={0 0}, upload=359886]: Begin upload segment... Dec 22 17:25:17.560654236 (409h56m17.431737271s) [seg_pos={0 0}, upload=359886]: Done begin upload segment. Dec 22 17:25:17.560690104 (409h56m17.431773149s) [noise=true , address=145.79.227.17:4972 , piece_id=V73WJ7IBPOSLURVGCMPEIXDPFMSR2TZPLILDIORCZQW2AR5TNPHQ, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.560856350 (409h56m17.431939405s) [noise=true , address=45.91.116.8:4047 , piece_id=6VLBDQOMGQMTUAJZ6A4HN4YXDKJUASQDC7TW75YCX2UIRRG3XCLA, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.561528851 (409h56m17.432611886s) [noise=true , address=45.91.116.7:4010 , piece_id=NTPUBYB6TL2CXQ6ORDYL5WQTX5757X5OMARDVBYQUUXDM7GP2FFQ, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.561757606 (409h56m17.432840641s) [noise=true , address=145.79.227.43:4157 , piece_id=WTAZBGNS2HJKQEVFZQLKFEGIYBKWU6R56N5VPREV4XGHV2BRCY5Q, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.562057427 (409h56m17.433140472s) [noise=true , address=77.90.31.14:4663 , piece_id=A5JHRTLNP4O46MRP2IYUIZP5OZT34XFFJMQ3BT7NM6QD6N5FO4IA, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.562120377 (409h56m17.433203422s) [noise=true , address=145.79.227.29:4941 , piece_id=CXRKQRCSWWPE7CQSE3HJCJK66MXHZKJSXGYQHEJS5ENINGGCRB4Q, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.562425908 (409h56m17.433508943s) [noise=true , address=145.79.227.8:4329 , piece_id=O64T7V5GVK72U7HVHZZGMOUBT6UUA3266G27PH2WIV3OLLHOT5HA, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.562643784 (409h56m17.433726829s) [noise=true , address=45.91.116.14:4189 , piece_id=RKQF2MH3PS2IS5IY7NHBSYLHEYOFSBGAWBW64O6YEFERZTKNGT5A, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.563153915 (409h56m17.43423695s) [noise=true , address=145.79.227.35:4483 , piece_id=CLGUNTM5AXARDV3CGSCEX4LKKC4FNBOKL4AXS67RT4TTRXKHIXVQ, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.563168844 (409h56m17.434251889s) [noise=true , address=45.91.116.6:4976 , piece_id=MYTSQNSERRFIMLUSKBV663OQ7VFXNQC5GRPVQZNXNXRXC2WGHWCA, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.563482751 (409h56m17.434565786s) [noise=true , address=145.79.227.36:4518 , piece_id=HADDUUSU4SDACSKXERKPFCBD62PBFHJKS2UYTI4PLTQPBV5GV7XQ, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.563558726 (409h56m17.434641771s) [noise=true , address=145.79.227.9:4365 , piece_id=3SURH75HZRY46HAE77BTWOL7SA4W5QZEIQ66LOK2HETQIOXJUEGQ, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.563893514 (409h56m17.434976549s) [noise=true , address=145.79.227.42:4123 , piece_id=ZOVGDENWTEY43GURA5B5ON4IE4IMRWFUDZ45MROTQM3HYQJB6HLA, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.564035464 (409h56m17.435118499s) [noise=true , address=145.79.227.39:4015 , piece_id=JAZP2LRDWZBPIK5NYD2ZP4LLAA2RAI4VOVV7GNVXHT5IR6XQ4DQA, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.564201710 (409h56m17.435284745s) [noise=true , address=145.79.227.20:4438 , piece_id=6U3VTJDLTODULTIJBG5Y5SPZTY3YKTZI2NUJ7KDWO4M5MROOZGBQ, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:17.564622482 (409h56m17.435705527s) [noise=true , address=145.79.227.46:4603 , piece_id=DFSBAFC7AFDPHGTESOUYR5DPNHABWLBXOMNWL5Y5BMD7C6GEIWQQ, seg_pos={0 0}, upload=359886]: Uploading piece... Dec 22 17:25:19.449351158 (409h56m19.320434203s) [upload=359886]: Next returned nil segment Dec 22 17:25:19.449370636 (409h56m19.320453681s) [upload=359886]: Waiting for error group managing segments... Dec 22 17:25:19.748564339 (409h56m19.619647374s) [noise=true , address=77.90.31.14:4663 , piece_id=A5JHRTLNP4O46MRP2IYUIZP5OZT34XFFJMQ3BT7NM6QD6N5FO4IA, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:19.805746453 (409h56m19.676829498s) [noise=true , address=45.91.116.6:4976 , piece_id=MYTSQNSERRFIMLUSKBV663OQ7VFXNQC5GRPVQZNXNXRXC2WGHWCA, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:19.851194944 (409h56m19.722277979s) [noise=true , address=45.91.116.14:4189 , piece_id=RKQF2MH3PS2IS5IY7NHBSYLHEYOFSBGAWBW64O6YEFERZTKNGT5A, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:20.258877556 (409h56m20.129960591s) [noise=true , address=45.91.116.7:4010 , piece_id=NTPUBYB6TL2CXQ6ORDYL5WQTX5757X5OMARDVBYQUUXDM7GP2FFQ, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:20.501207282 (409h56m20.372290317s) [noise=true , address=145.79.227.9:4365 , piece_id=3SURH75HZRY46HAE77BTWOL7SA4W5QZEIQ66LOK2HETQIOXJUEGQ, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:20.503600950 (409h56m20.374683995s) [noise=true , address=145.79.227.20:4438 , piece_id=6U3VTJDLTODULTIJBG5Y5SPZTY3YKTZI2NUJ7KDWO4M5MROOZGBQ, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:20.503819566 (409h56m20.374902621s) [noise=true , address=145.79.227.42:4123 , piece_id=ZOVGDENWTEY43GURA5B5ON4IE4IMRWFUDZ45MROTQM3HYQJB6HLA, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:20.504785966 (409h56m20.375869011s) [noise=true , address=145.79.227.17:4972 , piece_id=V73WJ7IBPOSLURVGCMPEIXDPFMSR2TZPLILDIORCZQW2AR5TNPHQ, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:20.513465874 (409h56m20.384548919s) [noise=true , address=145.79.227.43:4157 , piece_id=WTAZBGNS2HJKQEVFZQLKFEGIYBKWU6R56N5VPREV4XGHV2BRCY5Q, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:20.529535893 (409h56m20.400618939s) [noise=true , address=145.79.227.39:4015 , piece_id=JAZP2LRDWZBPIK5NYD2ZP4LLAA2RAI4VOVV7GNVXHT5IR6XQ4DQA, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:20.540295722 (409h56m20.411378767s) [noise=true , address=145.79.227.36:4518 , piece_id=HADDUUSU4SDACSKXERKPFCBD62PBFHJKS2UYTI4PLTQPBV5GV7XQ, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:20.543942486 (409h56m20.415025521s) [noise=true , address=145.79.227.46:4603 , piece_id=DFSBAFC7AFDPHGTESOUYR5DPNHABWLBXOMNWL5Y5BMD7C6GEIWQQ, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:20.547119405 (409h56m20.41820245s) [noise=true , address=145.79.227.8:4329 , piece_id=O64T7V5GVK72U7HVHZZGMOUBT6UUA3266G27PH2WIV3OLLHOT5HA, seg_pos={0 0}, upload=359886]: Done uploading piece. err: <nil> Dec 22 17:25:20.547141568 (409h56m20.418224623s) [seg_pos={0 0}, upload=359886]: Segment reached optimal threshold of 13 pieces. Dec 22 17:25:20.547358000 (409h56m20.418441045s) [noise=true , address=45.91.116.8:4047 , piece_id=6VLBDQOMGQMTUAJZ6A4HN4YXDKJUASQDC7TW75YCX2UIRRG3XCLA, seg_pos={0 0}, upload=359886]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 22 17:25:20.547365183 (409h56m20.418448228s) [noise=true , address=45.91.116.8:4047 , piece_id=6VLBDQOMGQMTUAJZ6A4HN4YXDKJUASQDC7TW75YCX2UIRRG3XCLA, seg_pos={0 0}, upload=359886]: Long tail cancellation due to optimal threshold Dec 22 17:25:20.547490342 (409h56m20.418573377s) [noise=true , address=145.79.227.35:4483 , piece_id=CLGUNTM5AXARDV3CGSCEX4LKKC4FNBOKL4AXS67RT4TTRXKHIXVQ, seg_pos={0 0}, upload=359886]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 22 17:25:20.547499048 (409h56m20.418582103s) [noise=true , address=145.79.227.35:4483 , piece_id=CLGUNTM5AXARDV3CGSCEX4LKKC4FNBOKL4AXS67RT4TTRXKHIXVQ, seg_pos={0 0}, upload=359886]: Long tail cancellation due to optimal threshold Dec 22 17:25:20.547819959 (409h56m20.418902994s) [noise=true , address=145.79.227.29:4941 , piece_id=CXRKQRCSWWPE7CQSE3HJCJK66MXHZKJSXGYQHEJS5ENINGGCRB4Q, seg_pos={0 0}, upload=359886]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 22 17:25:20.547838425 (409h56m20.41892147s) [noise=true , address=145.79.227.29:4941 , piece_id=CXRKQRCSWWPE7CQSE3HJCJK66MXHZKJSXGYQHEJS5ENINGGCRB4Q, seg_pos={0 0}, upload=359886]: Long tail cancellation due to optimal threshold Dec 22 17:25:20.547891726 (409h56m20.418974771s) [seg_pos={0 0}, upload=359886]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:25:20.547995043 (409h56m20.419078088s) [upload=359886]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:25:20.548001636 (409h56m20.419084681s) [upload=359886]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:25:20.838049653 (409h56m20.709132708s) [upload=359886]: Done uploading segments... Dec 22 17:25:20.838071214 (409h56m20.709154269s) [upload=359886]: Upload finished. err: <nil> Duration: 7.444894319s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)