Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......X] duration: 30.003075515s trace: https://jaeger.dev.storj.io/trace/7fa74dcd6777d738 FAIL (2025-12-26 17:09:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 2V35BJ3SDKMY3GWKTC6MGMPDPKSXQGUYV4OQ2SE747T6KCMSCIWA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: MNHP3ICGPDFXB5CBBVWQHF365GOQPST46JDUBI6BT2UGWRFK6WOQ) context deadline exceeded ======LOGS====== Dec 26 17:09:29.133151177 (505h40m29.004234222s) [upload=443729]: Starting upload Dec 26 17:09:29.133227001 (505h40m29.004310056s) [upload=443729]: Done starting upload Dec 26 17:09:29.133312353 (505h40m29.004395388s) [upload=443729]: Uploading segments... Dec 26 17:09:29.133467960 (505h40m29.004551005s) [upload=443729]: Got next segment. Inline: false Dec 26 17:09:29.133505902 (505h40m29.004588937s) [upload=443729]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:09:29.133511061 (505h40m29.004594117s) [upload=443729]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:09:29.385162329 (505h40m29.256245364s) [seg_pos={0 0}, upload=443729]: Begin upload segment... Dec 26 17:09:29.385217574 (505h40m29.256300609s) [seg_pos={0 0}, upload=443729]: Done begin upload segment. Dec 26 17:09:29.385297316 (505h40m29.256380361s) [noise=true , address=145.79.227.49:4338 , piece_id=R5JMKO5GTTPVZEU2NF2KCTQHVSRBJHLEA4XLO2WALKTWSHZ25TZQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.385311804 (505h40m29.256394839s) [noise=true , address=145.79.227.5:4222 , piece_id=SM5EUHIF3TUKSNTJFYIAPJX3GK56PHCPNM6BYNYFQIU5AXO6VOOA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386019221 (505h40m29.257102266s) [noise=true , address=77.90.31.15:4700 , piece_id=MNHP3ICGPDFXB5CBBVWQHF365GOQPST46JDUBI6BT2UGWRFK6WOQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386171460 (505h40m29.257254505s) [noise=true , address=145.79.227.52:4446 , piece_id=267V44INJDC35GFAVI3SF3ABEXZ2EEWCBKZNB2BMXXUZPA3CBUKA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386581772 (505h40m29.257664807s) [noise=true , address=145.79.227.42:4122 , piece_id=ND6SE7IYN2I7MEXE3XWY4TLHVR62OV6YMUD7RFTM4QWME4FVJYNQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386619003 (505h40m29.257702038s) [noise=true , address=145.79.227.38:4977 , piece_id=5HZR7ZNURSLY2WIRTL4V7QODZT4RCAX2A2EW6QG5OEJ72AKGJOHQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386876864 (505h40m29.257959899s) [noise=true , address=145.79.227.3:4149 , piece_id=UK5YDQW3ISQ4DGUVICJXSHWQYOMVD2XTGFULMAD5HGZS7M36TIHA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.387152179 (505h40m29.258235214s) [noise=true , address=45.91.116.6:4974 , piece_id=2V35BJ3SDKMY3GWKTC6MGMPDPKSXQGUYV4OQ2SE747T6KCMSCIWA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.387508908 (505h40m29.258591953s) [noise=true , address=145.79.227.47:4266 , piece_id=CV7AYDKW2QU36URGSMOUIM6JO4EU7AJA7HB2WQWXQ3Q4NYAX3WKQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.387793720 (505h40m29.258876755s) [noise=true , address=145.79.227.46:4601 , piece_id=GGQ7LNBL2C2YUD42SBTTNDKTTAYTJI2VMD3E35F2HNNSPVOPC46A, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.388066008 (505h40m29.259149054s) [noise=true , address=45.91.116.8:4047 , piece_id=VIW572WRWPYFXDDQIMODBEC676X6TRKHL4X6TK332MMS3X5E2Q4A, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.388185926 (505h40m29.259268971s) [noise=true , address=145.79.227.50:4374 , piece_id=P6WZAIBQQWA535GX4SGGPCJC6HDZYHMM4JBOXWFWPYZRYRLXJHLQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.392835450 (505h40m29.263918485s) [noise=true , address=77.90.31.14:4663 , piece_id=GVKAGH6B37ZXF5C5ITQZO726KRGDWTD4W3KFCLYQD6JKKNROOIEQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.393323039 (505h40m29.264406074s) [noise=true , address=45.91.116.12:4228 , piece_id=BDKXMPTAJSLEMKHRVX5QNRE6H57URA3JSJILN2RWQ5GFFKAD6GAQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.394641019 (505h40m29.265724064s) [noise=true , address=145.79.227.12:4767 , piece_id=SPVO53RPSN3JFWIG4HYD2BB4XIZ2F7AAGJYA7EYYJWDDN22LDOIQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.395598301 (505h40m29.266681346s) [noise=true , address=45.91.116.14:4190 , piece_id=LA7AEL6DNGQLBQHVTF7SGMQLVZNF5BOYY7EOZIPIC5YSNIUHXQ3Q, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:31.005665530 (505h40m30.876748575s) [upload=443729]: Next returned nil segment Dec 26 17:09:31.005677944 (505h40m30.876760989s) [upload=443729]: Waiting for error group managing segments... Dec 26 17:09:31.389076481 (505h40m31.260159516s) [noise=true , address=145.79.227.52:4446 , piece_id=267V44INJDC35GFAVI3SF3ABEXZ2EEWCBKZNB2BMXXUZPA3CBUKA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.391690869 (505h40m31.262773904s) [noise=true , address=145.79.227.46:4601 , piece_id=GGQ7LNBL2C2YUD42SBTTNDKTTAYTJI2VMD3E35F2HNNSPVOPC46A, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.396665723 (505h40m31.267748768s) [noise=true , address=145.79.227.5:4222 , piece_id=SM5EUHIF3TUKSNTJFYIAPJX3GK56PHCPNM6BYNYFQIU5AXO6VOOA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.398209442 (505h40m31.269292487s) [noise=true , address=145.79.227.49:4338 , piece_id=R5JMKO5GTTPVZEU2NF2KCTQHVSRBJHLEA4XLO2WALKTWSHZ25TZQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.400697820 (505h40m31.271780866s) [noise=true , address=145.79.227.38:4977 , piece_id=5HZR7ZNURSLY2WIRTL4V7QODZT4RCAX2A2EW6QG5OEJ72AKGJOHQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.408174808 (505h40m31.279257853s) [noise=true , address=145.79.227.50:4374 , piece_id=P6WZAIBQQWA535GX4SGGPCJC6HDZYHMM4JBOXWFWPYZRYRLXJHLQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.409061566 (505h40m31.280144611s) [noise=true , address=145.79.227.42:4122 , piece_id=ND6SE7IYN2I7MEXE3XWY4TLHVR62OV6YMUD7RFTM4QWME4FVJYNQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.417519532 (505h40m31.288602567s) [noise=true , address=145.79.227.3:4149 , piece_id=UK5YDQW3ISQ4DGUVICJXSHWQYOMVD2XTGFULMAD5HGZS7M36TIHA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.424664949 (505h40m31.295747984s) [noise=true , address=145.79.227.47:4266 , piece_id=CV7AYDKW2QU36URGSMOUIM6JO4EU7AJA7HB2WQWXQ3Q4NYAX3WKQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.425492514 (505h40m31.296575549s) [noise=true , address=145.79.227.12:4767 , piece_id=SPVO53RPSN3JFWIG4HYD2BB4XIZ2F7AAGJYA7EYYJWDDN22LDOIQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.664797552 (505h40m31.535880597s) [noise=true , address=45.91.116.6:4974 , piece_id=2V35BJ3SDKMY3GWKTC6MGMPDPKSXQGUYV4OQ2SE747T6KCMSCIWA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.682185633 (505h40m31.553268668s) [noise=true , address=77.90.31.14:4663 , piece_id=GVKAGH6B37ZXF5C5ITQZO726KRGDWTD4W3KFCLYQD6JKKNROOIEQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.706708498 (505h40m31.577791543s) [noise=true , address=77.90.31.15:4700 , piece_id=MNHP3ICGPDFXB5CBBVWQHF365GOQPST46JDUBI6BT2UGWRFK6WOQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.706727965 (505h40m31.577811011s) [seg_pos={0 0}, upload=443729]: Segment reached optimal threshold of 13 pieces. Dec 26 17:09:31.706934148 (505h40m31.578017193s) [noise=true , address=45.91.116.8:4047 , piece_id=VIW572WRWPYFXDDQIMODBEC676X6TRKHL4X6TK332MMS3X5E2Q4A, seg_pos={0 0}, upload=443729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 17:09:31.706941081 (505h40m31.578024116s) [noise=true , address=45.91.116.8:4047 , piece_id=VIW572WRWPYFXDDQIMODBEC676X6TRKHL4X6TK332MMS3X5E2Q4A, seg_pos={0 0}, upload=443729]: Long tail cancellation due to optimal threshold Dec 26 17:09:31.707030611 (505h40m31.578113646s) [noise=true , address=45.91.116.14:4190 , piece_id=LA7AEL6DNGQLBQHVTF7SGMQLVZNF5BOYY7EOZIPIC5YSNIUHXQ3Q, seg_pos={0 0}, upload=443729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 17:09:31.707038447 (505h40m31.578121482s) [noise=true , address=45.91.116.14:4190 , piece_id=LA7AEL6DNGQLBQHVTF7SGMQLVZNF5BOYY7EOZIPIC5YSNIUHXQ3Q, seg_pos={0 0}, upload=443729]: Long tail cancellation due to optimal threshold Dec 26 17:09:31.707187059 (505h40m31.578270105s) [noise=true , address=45.91.116.12:4228 , piece_id=BDKXMPTAJSLEMKHRVX5QNRE6H57URA3JSJILN2RWQ5GFFKAD6GAQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 17:09:31.707193411 (505h40m31.578276446s) [noise=true , address=45.91.116.12:4228 , piece_id=BDKXMPTAJSLEMKHRVX5QNRE6H57URA3JSJILN2RWQ5GFFKAD6GAQ, seg_pos={0 0}, upload=443729]: Long tail cancellation due to optimal threshold Dec 26 17:09:31.707217648 (505h40m31.578300683s) [seg_pos={0 0}, upload=443729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:09:31.707296969 (505h40m31.578380004s) [upload=443729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:09:31.707301958 (505h40m31.578384993s) [upload=443729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:09:32.015660245 (505h40m31.88674329s) [upload=443729]: Done uploading segments... Dec 26 17:09:32.015718627 (505h40m31.886801662s) [upload=443729]: Upload finished. err: <nil> Duration: 30.003086706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......] duration: 28.674908803s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:09:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 28.674937548s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X......XXX...X........X..............................] duration: 3.430116345s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:09:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430143326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X...............X.......X....X..........................] duration: 30.634200625s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:08:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 30.63425041s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........X......XXX...X........X.............................] duration: 3.452427401s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:08:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.452454311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X.....] duration: 7.473619703s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:08:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 7.473647777s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........X......XXX...X........X............................] duration: 3.490434331s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:08:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490467945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 7.724234689s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:07:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.724258975s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........X......XXX...X........X...........................] duration: 3.545993906s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:07:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546018764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 21.472964818s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:07:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 21.472995765s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X...............X.......X....X.........................] duration: 30.321610847s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:07:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 30.321640295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X....] duration: 6.126098129s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:07:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 6.126131202s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.139869508s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:07:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.139898913s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..X...............X.......X....X........................] duration: 29.326064837s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:07:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 29.326100797s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X..........X......XXX...X........X..........................] duration: 3.671401743s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:06:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.671448954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X...] duration: 6.172151534s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:06:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 6.172179206s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XX..........X......XXX...X........X.........................] duration: 3.914215043s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:06:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.91434957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 8.704370947s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:06:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 8.704395354s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..X...............X.......X....X.......................] duration: 23.091033912s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:06:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 23.091054021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.XX..........X......XXX...X........X........................] duration: 3.469748642s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:05:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469770463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..X...............X.......X....X......................] duration: 48.536851827s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:05:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 48.536879771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 9.497002584s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:05:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 9.497018644s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X..] duration: 5.954588957s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:05:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 5.954608194s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 7.693761601s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:05:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.693792992s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X.] duration: 25.868765259s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:05:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 25.868782752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..X...............X.......X....X.....................] duration: 35.523815844s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:04:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 35.523841952s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..XX..........X......XXX...X........X.......................] duration: 4.297019848s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:04:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.297046399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X] duration: 30.002701801s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:04:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 30.002735284s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...XX..........X......XXX...X........X......................] duration: 3.462098813s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:04:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462119362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 7.162924166s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:04:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.162955346s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..X...............X.......X....X....................] duration: 38.902821399s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:03:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 38.902840134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....XX..........X......XXX...X........X.....................] duration: 3.531058065s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:03:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531073324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.] duration: 16.409813192s trace: https://jaeger.dev.storj.io/trace/47a12fef265fe8e2 FAIL (2025-12-26 17:03:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: ANLKYPF5DXUQX3V7TOCJR2OSYZFN5XI7UAEBL37YKS346XOPHXDQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: JW7ENELF4G6AD5FIZKLB3XQGCJVLM5DV7CWCS5QXOHF46Z35YESA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7WQX56GLWRG6PZFP3DNF5COF6RYT52PHL3NTQOE5MJFL64BH62XQ) context deadline exceeded ======LOGS====== s... Dec 26 17:02:56.372426380 (505h33m56.243509415s) [upload=443629]: Got next segment. Inline: false Dec 26 17:02:56.372539757 (505h33m56.243622802s) [upload=443629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:02:56.372546510 (505h33m56.243629545s) [upload=443629]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:02:56.618142927 (505h33m56.489225962s) [seg_pos={0 0}, upload=443629]: Begin upload segment... Dec 26 17:02:56.618218431 (505h33m56.489301476s) [seg_pos={0 0}, upload=443629]: Done begin upload segment. Dec 26 17:02:56.618281171 (505h33m56.489364206s) [noise=true , address=145.79.227.16:4912 , piece_id=5RO23B7ROXA7GZOKRQRARCNBRPKRZOQPD46B6CEDCW4DX5JUJJ4Q, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.619477068 (505h33m56.490560113s) [noise=true , address=145.79.227.29:4943 , piece_id=E35LVN5VXI4ALAYZCRHTURLMVC6YDAA6HUGRPA666CYKP4X3GPRA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.620043326 (505h33m56.491126382s) [noise=true , address=77.90.31.13:4629 , piece_id=ANLKYPF5DXUQX3V7TOCJR2OSYZFN5XI7UAEBL37YKS346XOPHXDQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.620486910 (505h33m56.491569955s) [noise=true , address=145.79.227.47:4265 , piece_id=HQOUAJFSOSD5ZZ7ZXMHWHIP5ATQFZXB5PJIJT6YLPJAH6AVHCMEA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.620834462 (505h33m56.491917497s) [noise=true , address=45.91.116.8:4045 , piece_id=JW7ENELF4G6AD5FIZKLB3XQGCJVLM5DV7CWCS5QXOHF46Z35YESA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.621178037 (505h33m56.492261082s) [noise=true , address=145.79.227.50:4374 , piece_id=2M2PILEN5KZAOFYXTTUI45FUOJIHACC6KOG2Q6REHKDTL6GODPJQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.621557860 (505h33m56.492640905s) [noise=true , address=45.91.116.13:4156 , piece_id=UTKKKGDUBOSBVTNX7HW2COHTTQABO7AOUDZNRBT4TWKRJWEL2MZA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.622087077 (505h33m56.493170123s) [noise=true , address=145.79.227.46:4603 , piece_id=XVI7WCCU5AG7IIRTY5M3UOERXRLUOYAQY3U2A7DQIPAOIDUX4VTA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.622512839 (505h33m56.493595884s) [noise=true , address=145.79.227.48:4301 , piece_id=MFE2O3E3RSHH7NMHPQB3MLPW4XA4NNNPYW3KABDHAFATHV4QNVTA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.622864158 (505h33m56.493947203s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVNHVUYNWHHBR2J4LTOHN4MOVH2DDX4DKWBPIHEWLZ5YHMJWNNYQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.623228782 (505h33m56.494311827s) [noise=true , address=145.79.227.45:4230 , piece_id=Z3AFJ67RIY4D7O5L5C7G26PQ7G3LHGAJJR36PEHT53TWSTGTSQBQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.623620557 (505h33m56.494703602s) [noise=true , address=77.90.31.10:4517 , piece_id=ALL5WRUOC7QSI2ZZ44JBMM5BICSKFRF5KDGTFXS6EKPOGJSN45SA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.624010571 (505h33m56.495093616s) [noise=true , address=45.91.116.5:4299 , piece_id=7WQX56GLWRG6PZFP3DNF5COF6RYT52PHL3NTQOE5MJFL64BH62XQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.624399321 (505h33m56.495482366s) [noise=true , address=145.79.227.22:4509 , piece_id=JX27J4R3XKYKR3LDIETJJ5USAWREA3CBZ5SDERD2PMLDJVDBDLYA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.624816255 (505h33m56.49589929s) [noise=true , address=145.79.227.2:4114 , piece_id=RCKBOYQDY5ROW7ODK7ADVGWPT2MM753Z37V2SDDZDYG32NKC4FVQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.625131285 (505h33m56.49621432s) [noise=true , address=145.79.227.34:4905 , piece_id=KJRFUYCMHG3DGO33GAKGVK4KRSLSPR7WFAB5QLYJZTUIZZ6IMK7A, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:58.327842451 (505h33m58.198925496s) [upload=443629]: Next returned nil segment Dec 26 17:02:58.327867940 (505h33m58.198950985s) [upload=443629]: Waiting for error group managing segments... Dec 26 17:02:58.628733781 (505h33m58.499816816s) [noise=true , address=145.79.227.45:4230 , piece_id=Z3AFJ67RIY4D7O5L5C7G26PQ7G3LHGAJJR36PEHT53TWSTGTSQBQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.634889603 (505h33m58.505972638s) [noise=true , address=145.79.227.22:4509 , piece_id=JX27J4R3XKYKR3LDIETJJ5USAWREA3CBZ5SDERD2PMLDJVDBDLYA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.636250083 (505h33m58.507333138s) [noise=true , address=145.79.227.50:4374 , piece_id=2M2PILEN5KZAOFYXTTUI45FUOJIHACC6KOG2Q6REHKDTL6GODPJQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.639680364 (505h33m58.51076341s) [noise=true , address=145.79.227.47:4265 , piece_id=HQOUAJFSOSD5ZZ7ZXMHWHIP5ATQFZXB5PJIJT6YLPJAH6AVHCMEA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.641408375 (505h33m58.51249142s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVNHVUYNWHHBR2J4LTOHN4MOVH2DDX4DKWBPIHEWLZ5YHMJWNNYQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.644045357 (505h33m58.515128392s) [noise=true , address=145.79.227.46:4603 , piece_id=XVI7WCCU5AG7IIRTY5M3UOERXRLUOYAQY3U2A7DQIPAOIDUX4VTA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.644273631 (505h33m58.515356676s) [noise=true , address=145.79.227.2:4114 , piece_id=RCKBOYQDY5ROW7ODK7ADVGWPT2MM753Z37V2SDDZDYG32NKC4FVQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.649689564 (505h33m58.520772609s) [noise=true , address=145.79.227.48:4301 , piece_id=MFE2O3E3RSHH7NMHPQB3MLPW4XA4NNNPYW3KABDHAFATHV4QNVTA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.727898707 (505h33m58.598981742s) [noise=true , address=145.79.227.34:4905 , piece_id=KJRFUYCMHG3DGO33GAKGVK4KRSLSPR7WFAB5QLYJZTUIZZ6IMK7A, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.923049338 (505h33m58.794132383s) [noise=true , address=77.90.31.13:4629 , piece_id=ANLKYPF5DXUQX3V7TOCJR2OSYZFN5XI7UAEBL37YKS346XOPHXDQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.019884739 (505h33m58.890967784s) [noise=true , address=77.90.31.10:4517 , piece_id=ALL5WRUOC7QSI2ZZ44JBMM5BICSKFRF5KDGTFXS6EKPOGJSN45SA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.047118767 (505h33m58.918201802s) [noise=true , address=45.91.116.5:4299 , piece_id=7WQX56GLWRG6PZFP3DNF5COF6RYT52PHL3NTQOE5MJFL64BH62XQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.079095427 (505h33m58.950178462s) [noise=true , address=45.91.116.8:4045 , piece_id=JW7ENELF4G6AD5FIZKLB3XQGCJVLM5DV7CWCS5QXOHF46Z35YESA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.079114894 (505h33m58.950197929s) [seg_pos={0 0}, upload=443629]: Segment reached optimal threshold of 13 pieces. Dec 26 17:02:59.079288835 (505h33m58.95037188s) [noise=true , address=145.79.227.29:4943 , piece_id=E35LVN5VXI4ALAYZCRHTURLMVC6YDAA6HUGRPA666CYKP4X3GPRA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:02:59.079297371 (505h33m58.950380416s) [noise=true , address=145.79.227.29:4943 , piece_id=E35LVN5VXI4ALAYZCRHTURLMVC6YDAA6HUGRPA666CYKP4X3GPRA, seg_pos={0 0}, upload=443629]: Long tail cancellation due to optimal threshold Dec 26 17:02:59.079433621 (505h33m58.950516676s) [noise=true , address=145.79.227.16:4912 , piece_id=5RO23B7ROXA7GZOKRQRARCNBRPKRZOQPD46B6CEDCW4DX5JUJJ4Q, seg_pos={0 0}, upload=443629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:33400->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:33400->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33400->145.79.227.16:4912: use of closed network connection Dec 26 17:02:59.079449230 (505h33m58.950532265s) [noise=true , address=145.79.227.16:4912 , piece_id=5RO23B7ROXA7GZOKRQRARCNBRPKRZOQPD46B6CEDCW4DX5JUJJ4Q, seg_pos={0 0}, upload=443629]: Long tail cancellation due to optimal threshold Dec 26 17:02:59.079559660 (505h33m58.950642705s) [noise=true , address=45.91.116.13:4156 , piece_id=UTKKKGDUBOSBVTNX7HW2COHTTQABO7AOUDZNRBT4TWKRJWEL2MZA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:45836->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:45836->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45836->45.91.116.13:4156: use of closed network connection Dec 26 17:02:59.079569590 (505h33m58.950652625s) [noise=true , address=45.91.116.13:4156 , piece_id=UTKKKGDUBOSBVTNX7HW2COHTTQABO7AOUDZNRBT4TWKRJWEL2MZA, seg_pos={0 0}, upload=443629]: Long tail cancellation due to optimal threshold Dec 26 17:02:59.079589898 (505h33m58.950672933s) [seg_pos={0 0}, upload=443629]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:02:59.079665502 (505h33m58.950748537s) [upload=443629]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:02:59.079671474 (505h33m58.950754509s) [upload=443629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:02:59.375961150 (505h33m59.247044195s) [upload=443629]: Done uploading segments... Dec 26 17:02:59.375986308 (505h33m59.247069343s) [upload=443629]: Upload finished. err: <nil> Duration: 16.409830254s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....XX..........X......XXX...X........X....................] duration: 3.502484356s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:03:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502511577s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.XX.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X] duration: 30.002740646s trace: https://jaeger.dev.storj.io/trace/47a12fef265fe8e2 FAIL (2025-12-26 17:03:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: ANLKYPF5DXUQX3V7TOCJR2OSYZFN5XI7UAEBL37YKS346XOPHXDQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: JW7ENELF4G6AD5FIZKLB3XQGCJVLM5DV7CWCS5QXOHF46Z35YESA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7WQX56GLWRG6PZFP3DNF5COF6RYT52PHL3NTQOE5MJFL64BH62XQ) context deadline exceeded ======LOGS====== s... Dec 26 17:02:56.372426380 (505h33m56.243509415s) [upload=443629]: Got next segment. Inline: false Dec 26 17:02:56.372539757 (505h33m56.243622802s) [upload=443629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:02:56.372546510 (505h33m56.243629545s) [upload=443629]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:02:56.618142927 (505h33m56.489225962s) [seg_pos={0 0}, upload=443629]: Begin upload segment... Dec 26 17:02:56.618218431 (505h33m56.489301476s) [seg_pos={0 0}, upload=443629]: Done begin upload segment. Dec 26 17:02:56.618281171 (505h33m56.489364206s) [noise=true , address=145.79.227.16:4912 , piece_id=5RO23B7ROXA7GZOKRQRARCNBRPKRZOQPD46B6CEDCW4DX5JUJJ4Q, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.619477068 (505h33m56.490560113s) [noise=true , address=145.79.227.29:4943 , piece_id=E35LVN5VXI4ALAYZCRHTURLMVC6YDAA6HUGRPA666CYKP4X3GPRA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.620043326 (505h33m56.491126382s) [noise=true , address=77.90.31.13:4629 , piece_id=ANLKYPF5DXUQX3V7TOCJR2OSYZFN5XI7UAEBL37YKS346XOPHXDQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.620486910 (505h33m56.491569955s) [noise=true , address=145.79.227.47:4265 , piece_id=HQOUAJFSOSD5ZZ7ZXMHWHIP5ATQFZXB5PJIJT6YLPJAH6AVHCMEA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.620834462 (505h33m56.491917497s) [noise=true , address=45.91.116.8:4045 , piece_id=JW7ENELF4G6AD5FIZKLB3XQGCJVLM5DV7CWCS5QXOHF46Z35YESA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.621178037 (505h33m56.492261082s) [noise=true , address=145.79.227.50:4374 , piece_id=2M2PILEN5KZAOFYXTTUI45FUOJIHACC6KOG2Q6REHKDTL6GODPJQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.621557860 (505h33m56.492640905s) [noise=true , address=45.91.116.13:4156 , piece_id=UTKKKGDUBOSBVTNX7HW2COHTTQABO7AOUDZNRBT4TWKRJWEL2MZA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.622087077 (505h33m56.493170123s) [noise=true , address=145.79.227.46:4603 , piece_id=XVI7WCCU5AG7IIRTY5M3UOERXRLUOYAQY3U2A7DQIPAOIDUX4VTA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.622512839 (505h33m56.493595884s) [noise=true , address=145.79.227.48:4301 , piece_id=MFE2O3E3RSHH7NMHPQB3MLPW4XA4NNNPYW3KABDHAFATHV4QNVTA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.622864158 (505h33m56.493947203s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVNHVUYNWHHBR2J4LTOHN4MOVH2DDX4DKWBPIHEWLZ5YHMJWNNYQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.623228782 (505h33m56.494311827s) [noise=true , address=145.79.227.45:4230 , piece_id=Z3AFJ67RIY4D7O5L5C7G26PQ7G3LHGAJJR36PEHT53TWSTGTSQBQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.623620557 (505h33m56.494703602s) [noise=true , address=77.90.31.10:4517 , piece_id=ALL5WRUOC7QSI2ZZ44JBMM5BICSKFRF5KDGTFXS6EKPOGJSN45SA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.624010571 (505h33m56.495093616s) [noise=true , address=45.91.116.5:4299 , piece_id=7WQX56GLWRG6PZFP3DNF5COF6RYT52PHL3NTQOE5MJFL64BH62XQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.624399321 (505h33m56.495482366s) [noise=true , address=145.79.227.22:4509 , piece_id=JX27J4R3XKYKR3LDIETJJ5USAWREA3CBZ5SDERD2PMLDJVDBDLYA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.624816255 (505h33m56.49589929s) [noise=true , address=145.79.227.2:4114 , piece_id=RCKBOYQDY5ROW7ODK7ADVGWPT2MM753Z37V2SDDZDYG32NKC4FVQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.625131285 (505h33m56.49621432s) [noise=true , address=145.79.227.34:4905 , piece_id=KJRFUYCMHG3DGO33GAKGVK4KRSLSPR7WFAB5QLYJZTUIZZ6IMK7A, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:58.327842451 (505h33m58.198925496s) [upload=443629]: Next returned nil segment Dec 26 17:02:58.327867940 (505h33m58.198950985s) [upload=443629]: Waiting for error group managing segments... Dec 26 17:02:58.628733781 (505h33m58.499816816s) [noise=true , address=145.79.227.45:4230 , piece_id=Z3AFJ67RIY4D7O5L5C7G26PQ7G3LHGAJJR36PEHT53TWSTGTSQBQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.634889603 (505h33m58.505972638s) [noise=true , address=145.79.227.22:4509 , piece_id=JX27J4R3XKYKR3LDIETJJ5USAWREA3CBZ5SDERD2PMLDJVDBDLYA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.636250083 (505h33m58.507333138s) [noise=true , address=145.79.227.50:4374 , piece_id=2M2PILEN5KZAOFYXTTUI45FUOJIHACC6KOG2Q6REHKDTL6GODPJQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.639680364 (505h33m58.51076341s) [noise=true , address=145.79.227.47:4265 , piece_id=HQOUAJFSOSD5ZZ7ZXMHWHIP5ATQFZXB5PJIJT6YLPJAH6AVHCMEA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.641408375 (505h33m58.51249142s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVNHVUYNWHHBR2J4LTOHN4MOVH2DDX4DKWBPIHEWLZ5YHMJWNNYQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.644045357 (505h33m58.515128392s) [noise=true , address=145.79.227.46:4603 , piece_id=XVI7WCCU5AG7IIRTY5M3UOERXRLUOYAQY3U2A7DQIPAOIDUX4VTA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.644273631 (505h33m58.515356676s) [noise=true , address=145.79.227.2:4114 , piece_id=RCKBOYQDY5ROW7ODK7ADVGWPT2MM753Z37V2SDDZDYG32NKC4FVQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.649689564 (505h33m58.520772609s) [noise=true , address=145.79.227.48:4301 , piece_id=MFE2O3E3RSHH7NMHPQB3MLPW4XA4NNNPYW3KABDHAFATHV4QNVTA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.727898707 (505h33m58.598981742s) [noise=true , address=145.79.227.34:4905 , piece_id=KJRFUYCMHG3DGO33GAKGVK4KRSLSPR7WFAB5QLYJZTUIZZ6IMK7A, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.923049338 (505h33m58.794132383s) [noise=true , address=77.90.31.13:4629 , piece_id=ANLKYPF5DXUQX3V7TOCJR2OSYZFN5XI7UAEBL37YKS346XOPHXDQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.019884739 (505h33m58.890967784s) [noise=true , address=77.90.31.10:4517 , piece_id=ALL5WRUOC7QSI2ZZ44JBMM5BICSKFRF5KDGTFXS6EKPOGJSN45SA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.047118767 (505h33m58.918201802s) [noise=true , address=45.91.116.5:4299 , piece_id=7WQX56GLWRG6PZFP3DNF5COF6RYT52PHL3NTQOE5MJFL64BH62XQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.079095427 (505h33m58.950178462s) [noise=true , address=45.91.116.8:4045 , piece_id=JW7ENELF4G6AD5FIZKLB3XQGCJVLM5DV7CWCS5QXOHF46Z35YESA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.079114894 (505h33m58.950197929s) [seg_pos={0 0}, upload=443629]: Segment reached optimal threshold of 13 pieces. Dec 26 17:02:59.079288835 (505h33m58.95037188s) [noise=true , address=145.79.227.29:4943 , piece_id=E35LVN5VXI4ALAYZCRHTURLMVC6YDAA6HUGRPA666CYKP4X3GPRA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:02:59.079297371 (505h33m58.950380416s) [noise=true , address=145.79.227.29:4943 , piece_id=E35LVN5VXI4ALAYZCRHTURLMVC6YDAA6HUGRPA666CYKP4X3GPRA, seg_pos={0 0}, upload=443629]: Long tail cancellation due to optimal threshold Dec 26 17:02:59.079433621 (505h33m58.950516676s) [noise=true , address=145.79.227.16:4912 , piece_id=5RO23B7ROXA7GZOKRQRARCNBRPKRZOQPD46B6CEDCW4DX5JUJJ4Q, seg_pos={0 0}, upload=443629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:33400->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:33400->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33400->145.79.227.16:4912: use of closed network connection Dec 26 17:02:59.079449230 (505h33m58.950532265s) [noise=true , address=145.79.227.16:4912 , piece_id=5RO23B7ROXA7GZOKRQRARCNBRPKRZOQPD46B6CEDCW4DX5JUJJ4Q, seg_pos={0 0}, upload=443629]: Long tail cancellation due to optimal threshold Dec 26 17:02:59.079559660 (505h33m58.950642705s) [noise=true , address=45.91.116.13:4156 , piece_id=UTKKKGDUBOSBVTNX7HW2COHTTQABO7AOUDZNRBT4TWKRJWEL2MZA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:45836->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:45836->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45836->45.91.116.13:4156: use of closed network connection Dec 26 17:02:59.079569590 (505h33m58.950652625s) [noise=true , address=45.91.116.13:4156 , piece_id=UTKKKGDUBOSBVTNX7HW2COHTTQABO7AOUDZNRBT4TWKRJWEL2MZA, seg_pos={0 0}, upload=443629]: Long tail cancellation due to optimal threshold Dec 26 17:02:59.079589898 (505h33m58.950672933s) [seg_pos={0 0}, upload=443629]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:02:59.079665502 (505h33m58.950748537s) [upload=443629]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:02:59.079671474 (505h33m58.950754509s) [upload=443629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:02:59.375961150 (505h33m59.247044195s) [upload=443629]: Done uploading segments... Dec 26 17:02:59.375986308 (505h33m59.247069343s) [upload=443629]: Upload finished. err: <nil> Duration: 30.002765674s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..X...............X.......X....X...................] duration: 44.46295264s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:03:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 44.462974681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 8.559787015s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:03:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 8.559815668s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 7.021733145s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:02:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.021764583s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X.....XX..........X......XXX...X........X...................] duration: 3.579845063s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:02:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.579874919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.XX.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..] duration: 5.951826715s trace: https://jaeger.dev.storj.io/trace/2ea6b7cc03d87efc FAIL (2025-12-26 17:02:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:00:41.074546763 (505h31m40.945629798s) [upload=443596]: Starting upload Dec 26 17:00:41.074674687 (505h31m40.945757732s) [upload=443596]: Done starting upload Dec 26 17:00:41.074739359 (505h31m40.945822404s) [upload=443596]: Uploading segments... Dec 26 17:00:41.074970079 (505h31m40.946053114s) [upload=443596]: Got next segment. Inline: false Dec 26 17:00:41.075003883 (505h31m40.946086929s) [upload=443596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:00:41.075008482 (505h31m40.946091517s) [upload=443596]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:00:41.322478708 (505h31m41.193561763s) [seg_pos={0 0}, upload=443596]: Begin upload segment... Dec 26 17:00:41.322545916 (505h31m41.193628961s) [seg_pos={0 0}, upload=443596]: Done begin upload segment. Dec 26 17:00:41.322605950 (505h31m41.193688985s) [noise=true , address=45.91.116.10:4084 , piece_id=L2IDPQS227IRJPUFCIPZICAJUNLZ2TGV7KICBAHBTG3JN66DKRNQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.322650866 (505h31m41.193733911s) [noise=true , address=145.79.227.17:4971 , piece_id=S2PMFG3NELSF7SFEIQTCYW4RY6AP62JET4JKAD5K4X6UP7I5WJKQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.323465316 (505h31m41.194548361s) [noise=true , address=77.90.31.11:4555 , piece_id=CKUTE7UJUE6NCN3TCX4MZVT3HWQTGEDVR2QQIFW4DDFVH7SIGA4A, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.324090377 (505h31m41.195173412s) [noise=true , address=145.79.227.39:4015 , piece_id=JRNT3EMKFDNR7NSKKBEZ4W73CTEQDHBDI2AN4SDC6Z5OLVEFVDZA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.324456204 (505h31m41.195539249s) [noise=true , address=145.79.227.41:4087 , piece_id=INHEWTEIW2YLVUJZ3RJE75SQZPGAM4LTRQKA4CYVOSK5XM7HSDRA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.324950565 (505h31m41.1960336s) [noise=true , address=145.79.227.22:4511 , piece_id=TNYHQSYXHHIRGGO3WU5ISZD4CKSEH6FZJITA7WO6PQGA2EDCY3WQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.325394541 (505h31m41.196477576s) [noise=true , address=45.91.116.6:4974 , piece_id=5P2OWJ74S3DDQ3CGYFL2KIAZUU3YO5TZMKQ2ZCBI24URYLR45REA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.325400442 (505h31m41.196483487s) [noise=true , address=145.79.227.45:4230 , piece_id=NCGLLCQ6G35KHW7RTR6RMWBCFGE643QMPWGPHMNLRRV2XO4AKJXA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.326139929 (505h31m41.197222975s) [noise=true , address=145.79.227.15:4876 , piece_id=MF63IBUO3K3OJJE53JVTUPGND6OA5SP7OSUGVVQ2TM3LNGTTBSMQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.326761233 (505h31m41.197844278s) [noise=true , address=145.79.227.52:4445 , piece_id=JNR2GDK4ZL32GH3L5QILVLFQMGNEHU4DC6X6DCFL5YH7IDKN5OZA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.327559563 (505h31m41.198642608s) [noise=true , address=45.91.116.13:4154 , piece_id=LBVYTFLJM6XVNXPPRBZX3577V5YG4AXXKW4UFAKBD32SEV326XDA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.328178341 (505h31m41.199261386s) [noise=true , address=145.79.227.37:4566 , piece_id=HJFAOPCO7BK3K6FIJHJGH6Y5HBTFXGFRCHW7BSAWYIUS2QMXZPQA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.330384672 (505h31m41.201467717s) [noise=true , address=145.79.227.2:4113 , piece_id=S3AUHP2P5LE4JIFSSXXA7LM4YB5VFHYNGZBXYTIEYDFNDS2ZI5IQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.332356868 (505h31m41.203439913s) [noise=true , address=45.91.116.14:4189 , piece_id=M4FYM3IRUT3N6BBCU7Z2RYC4YTEZ7QV3T5733BZU5SAKGQMDBP6Q, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.334755154 (505h31m41.205838199s) [noise=true , address=145.79.227.47:4266 , piece_id=5GOIJJ5KB2ED72P7CJ3CJKJRDI76D5NVWZ4BC2NAJZ5BJ4KT2ZTQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.336744082 (505h31m41.207827117s) [noise=true , address=145.79.227.11:4731 , piece_id=3UD2662V7S5FW2JAFAHWDWFHNZ3J6II4HLKMKBG6XIOYBTAUR33Q, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:42.487244964 (505h31m42.358328019s) [noise=true , address=77.90.31.11:4555 , piece_id=CKUTE7UJUE6NCN3TCX4MZVT3HWQTGEDVR2QQIFW4DDFVH7SIGA4A, seg_pos={0 0}, upload=443596]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: storage node overloaded, request limit: 2000 Dec 26 17:00:43.031756910 (505h31m42.902839945s) [upload=443596]: Next returned nil segment Dec 26 17:00:43.031782708 (505h31m42.902865753s) [upload=443596]: Waiting for error group managing segments... Dec 26 17:00:43.316897206 (505h31m43.187980251s) [noise=true , address=145.79.227.17:4971 , piece_id=S2PMFG3NELSF7SFEIQTCYW4RY6AP62JET4JKAD5K4X6UP7I5WJKQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.332085437 (505h31m43.203168492s) [noise=true , address=145.79.227.39:4015 , piece_id=JRNT3EMKFDNR7NSKKBEZ4W73CTEQDHBDI2AN4SDC6Z5OLVEFVDZA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.333057478 (505h31m43.204140513s) [noise=true , address=145.79.227.41:4087 , piece_id=INHEWTEIW2YLVUJZ3RJE75SQZPGAM4LTRQKA4CYVOSK5XM7HSDRA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.341314691 (505h31m43.212397726s) [noise=true , address=145.79.227.15:4876 , piece_id=MF63IBUO3K3OJJE53JVTUPGND6OA5SP7OSUGVVQ2TM3LNGTTBSMQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.343019046 (505h31m43.214102081s) [noise=true , address=145.79.227.45:4230 , piece_id=NCGLLCQ6G35KHW7RTR6RMWBCFGE643QMPWGPHMNLRRV2XO4AKJXA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.344413391 (505h31m43.215496436s) [noise=true , address=145.79.227.37:4566 , piece_id=HJFAOPCO7BK3K6FIJHJGH6Y5HBTFXGFRCHW7BSAWYIUS2QMXZPQA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.344517990 (505h31m43.215601025s) [noise=true , address=145.79.227.11:4731 , piece_id=3UD2662V7S5FW2JAFAHWDWFHNZ3J6II4HLKMKBG6XIOYBTAUR33Q, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.350207134 (505h31m43.221290179s) [noise=true , address=145.79.227.22:4511 , piece_id=TNYHQSYXHHIRGGO3WU5ISZD4CKSEH6FZJITA7WO6PQGA2EDCY3WQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.367585716 (505h31m43.238668751s) [noise=true , address=145.79.227.52:4445 , piece_id=JNR2GDK4ZL32GH3L5QILVLFQMGNEHU4DC6X6DCFL5YH7IDKN5OZA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.379997721 (505h31m43.251080756s) [noise=true , address=145.79.227.47:4266 , piece_id=5GOIJJ5KB2ED72P7CJ3CJKJRDI76D5NVWZ4BC2NAJZ5BJ4KT2ZTQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.388167787 (505h31m43.259250832s) [noise=true , address=145.79.227.2:4113 , piece_id=S3AUHP2P5LE4JIFSSXXA7LM4YB5VFHYNGZBXYTIEYDFNDS2ZI5IQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.698245359 (505h31m43.569328404s) [noise=true , address=45.91.116.10:4084 , piece_id=L2IDPQS227IRJPUFCIPZICAJUNLZ2TGV7KICBAHBTG3JN66DKRNQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:44.055826906 (505h31m43.926909941s) [noise=true , address=45.91.116.13:4154 , piece_id=LBVYTFLJM6XVNXPPRBZX3577V5YG4AXXKW4UFAKBD32SEV326XDA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:44.055845892 (505h31m43.926928927s) [seg_pos={0 0}, upload=443596]: Segment reached optimal threshold of 13 pieces. Dec 26 17:00:44.056062314 (505h31m43.927145349s) [noise=true , address=45.91.116.14:4189 , piece_id=M4FYM3IRUT3N6BBCU7Z2RYC4YTEZ7QV3T5733BZU5SAKGQMDBP6Q, seg_pos={0 0}, upload=443596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:45810->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:45810->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45810->45.91.116.14:4189: use of closed network connection Dec 26 17:00:44.056069407 (505h31m43.927152443s) [noise=true , address=45.91.116.14:4189 , piece_id=M4FYM3IRUT3N6BBCU7Z2RYC4YTEZ7QV3T5733BZU5SAKGQMDBP6Q, seg_pos={0 0}, upload=443596]: Long tail cancellation due to optimal threshold Dec 26 17:00:44.056471353 (505h31m43.927554398s) [noise=true , address=45.91.116.6:4974 , piece_id=5P2OWJ74S3DDQ3CGYFL2KIAZUU3YO5TZMKQ2ZCBI24URYLR45REA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 26 17:00:44.056482484 (505h31m43.927565519s) [noise=true , address=45.91.116.6:4974 , piece_id=5P2OWJ74S3DDQ3CGYFL2KIAZUU3YO5TZMKQ2ZCBI24URYLR45REA, seg_pos={0 0}, upload=443596]: Long tail cancellation due to optimal threshold Dec 26 17:00:44.056502382 (505h31m43.927585418s) [seg_pos={0 0}, upload=443596]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:00:44.056585761 (505h31m43.927668806s) [upload=443596]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:00:44.056592714 (505h31m43.927675749s) [upload=443596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:00:44.356920100 (505h31m44.228003135s) [upload=443596]: Done uploading segments... Dec 26 17:00:44.356939787 (505h31m44.228022822s) [upload=443596]: Upload finished. err: <nil> Duration: 5.951858266s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [XX.....XX..........X......XXX...X........X..................] duration: 3.479334977s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:02:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479367438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 7.134401866s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:02:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.134429379s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..X...............X.......X....X..................] duration: 23.279443106s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:02:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 23.279462914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.........X..X...............X.......X....X.................] duration: 42.698715858s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:01:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 42.698734344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X.XX.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX.] duration: 6.153124303s trace: https://jaeger.dev.storj.io/trace/2ea6b7cc03d87efc FAIL (2025-12-26 17:01:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:00:41.074546763 (505h31m40.945629798s) [upload=443596]: Starting upload Dec 26 17:00:41.074674687 (505h31m40.945757732s) [upload=443596]: Done starting upload Dec 26 17:00:41.074739359 (505h31m40.945822404s) [upload=443596]: Uploading segments... Dec 26 17:00:41.074970079 (505h31m40.946053114s) [upload=443596]: Got next segment. Inline: false Dec 26 17:00:41.075003883 (505h31m40.946086929s) [upload=443596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:00:41.075008482 (505h31m40.946091517s) [upload=443596]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:00:41.322478708 (505h31m41.193561763s) [seg_pos={0 0}, upload=443596]: Begin upload segment... Dec 26 17:00:41.322545916 (505h31m41.193628961s) [seg_pos={0 0}, upload=443596]: Done begin upload segment. Dec 26 17:00:41.322605950 (505h31m41.193688985s) [noise=true , address=45.91.116.10:4084 , piece_id=L2IDPQS227IRJPUFCIPZICAJUNLZ2TGV7KICBAHBTG3JN66DKRNQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.322650866 (505h31m41.193733911s) [noise=true , address=145.79.227.17:4971 , piece_id=S2PMFG3NELSF7SFEIQTCYW4RY6AP62JET4JKAD5K4X6UP7I5WJKQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.323465316 (505h31m41.194548361s) [noise=true , address=77.90.31.11:4555 , piece_id=CKUTE7UJUE6NCN3TCX4MZVT3HWQTGEDVR2QQIFW4DDFVH7SIGA4A, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.324090377 (505h31m41.195173412s) [noise=true , address=145.79.227.39:4015 , piece_id=JRNT3EMKFDNR7NSKKBEZ4W73CTEQDHBDI2AN4SDC6Z5OLVEFVDZA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.324456204 (505h31m41.195539249s) [noise=true , address=145.79.227.41:4087 , piece_id=INHEWTEIW2YLVUJZ3RJE75SQZPGAM4LTRQKA4CYVOSK5XM7HSDRA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.324950565 (505h31m41.1960336s) [noise=true , address=145.79.227.22:4511 , piece_id=TNYHQSYXHHIRGGO3WU5ISZD4CKSEH6FZJITA7WO6PQGA2EDCY3WQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.325394541 (505h31m41.196477576s) [noise=true , address=45.91.116.6:4974 , piece_id=5P2OWJ74S3DDQ3CGYFL2KIAZUU3YO5TZMKQ2ZCBI24URYLR45REA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.325400442 (505h31m41.196483487s) [noise=true , address=145.79.227.45:4230 , piece_id=NCGLLCQ6G35KHW7RTR6RMWBCFGE643QMPWGPHMNLRRV2XO4AKJXA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.326139929 (505h31m41.197222975s) [noise=true , address=145.79.227.15:4876 , piece_id=MF63IBUO3K3OJJE53JVTUPGND6OA5SP7OSUGVVQ2TM3LNGTTBSMQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.326761233 (505h31m41.197844278s) [noise=true , address=145.79.227.52:4445 , piece_id=JNR2GDK4ZL32GH3L5QILVLFQMGNEHU4DC6X6DCFL5YH7IDKN5OZA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.327559563 (505h31m41.198642608s) [noise=true , address=45.91.116.13:4154 , piece_id=LBVYTFLJM6XVNXPPRBZX3577V5YG4AXXKW4UFAKBD32SEV326XDA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.328178341 (505h31m41.199261386s) [noise=true , address=145.79.227.37:4566 , piece_id=HJFAOPCO7BK3K6FIJHJGH6Y5HBTFXGFRCHW7BSAWYIUS2QMXZPQA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.330384672 (505h31m41.201467717s) [noise=true , address=145.79.227.2:4113 , piece_id=S3AUHP2P5LE4JIFSSXXA7LM4YB5VFHYNGZBXYTIEYDFNDS2ZI5IQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.332356868 (505h31m41.203439913s) [noise=true , address=45.91.116.14:4189 , piece_id=M4FYM3IRUT3N6BBCU7Z2RYC4YTEZ7QV3T5733BZU5SAKGQMDBP6Q, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.334755154 (505h31m41.205838199s) [noise=true , address=145.79.227.47:4266 , piece_id=5GOIJJ5KB2ED72P7CJ3CJKJRDI76D5NVWZ4BC2NAJZ5BJ4KT2ZTQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.336744082 (505h31m41.207827117s) [noise=true , address=145.79.227.11:4731 , piece_id=3UD2662V7S5FW2JAFAHWDWFHNZ3J6II4HLKMKBG6XIOYBTAUR33Q, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:42.487244964 (505h31m42.358328019s) [noise=true , address=77.90.31.11:4555 , piece_id=CKUTE7UJUE6NCN3TCX4MZVT3HWQTGEDVR2QQIFW4DDFVH7SIGA4A, seg_pos={0 0}, upload=443596]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: storage node overloaded, request limit: 2000 Dec 26 17:00:43.031756910 (505h31m42.902839945s) [upload=443596]: Next returned nil segment Dec 26 17:00:43.031782708 (505h31m42.902865753s) [upload=443596]: Waiting for error group managing segments... Dec 26 17:00:43.316897206 (505h31m43.187980251s) [noise=true , address=145.79.227.17:4971 , piece_id=S2PMFG3NELSF7SFEIQTCYW4RY6AP62JET4JKAD5K4X6UP7I5WJKQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.332085437 (505h31m43.203168492s) [noise=true , address=145.79.227.39:4015 , piece_id=JRNT3EMKFDNR7NSKKBEZ4W73CTEQDHBDI2AN4SDC6Z5OLVEFVDZA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.333057478 (505h31m43.204140513s) [noise=true , address=145.79.227.41:4087 , piece_id=INHEWTEIW2YLVUJZ3RJE75SQZPGAM4LTRQKA4CYVOSK5XM7HSDRA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.341314691 (505h31m43.212397726s) [noise=true , address=145.79.227.15:4876 , piece_id=MF63IBUO3K3OJJE53JVTUPGND6OA5SP7OSUGVVQ2TM3LNGTTBSMQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.343019046 (505h31m43.214102081s) [noise=true , address=145.79.227.45:4230 , piece_id=NCGLLCQ6G35KHW7RTR6RMWBCFGE643QMPWGPHMNLRRV2XO4AKJXA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.344413391 (505h31m43.215496436s) [noise=true , address=145.79.227.37:4566 , piece_id=HJFAOPCO7BK3K6FIJHJGH6Y5HBTFXGFRCHW7BSAWYIUS2QMXZPQA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.344517990 (505h31m43.215601025s) [noise=true , address=145.79.227.11:4731 , piece_id=3UD2662V7S5FW2JAFAHWDWFHNZ3J6II4HLKMKBG6XIOYBTAUR33Q, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.350207134 (505h31m43.221290179s) [noise=true , address=145.79.227.22:4511 , piece_id=TNYHQSYXHHIRGGO3WU5ISZD4CKSEH6FZJITA7WO6PQGA2EDCY3WQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.367585716 (505h31m43.238668751s) [noise=true , address=145.79.227.52:4445 , piece_id=JNR2GDK4ZL32GH3L5QILVLFQMGNEHU4DC6X6DCFL5YH7IDKN5OZA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.379997721 (505h31m43.251080756s) [noise=true , address=145.79.227.47:4266 , piece_id=5GOIJJ5KB2ED72P7CJ3CJKJRDI76D5NVWZ4BC2NAJZ5BJ4KT2ZTQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.388167787 (505h31m43.259250832s) [noise=true , address=145.79.227.2:4113 , piece_id=S3AUHP2P5LE4JIFSSXXA7LM4YB5VFHYNGZBXYTIEYDFNDS2ZI5IQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.698245359 (505h31m43.569328404s) [noise=true , address=45.91.116.10:4084 , piece_id=L2IDPQS227IRJPUFCIPZICAJUNLZ2TGV7KICBAHBTG3JN66DKRNQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:44.055826906 (505h31m43.926909941s) [noise=true , address=45.91.116.13:4154 , piece_id=LBVYTFLJM6XVNXPPRBZX3577V5YG4AXXKW4UFAKBD32SEV326XDA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:44.055845892 (505h31m43.926928927s) [seg_pos={0 0}, upload=443596]: Segment reached optimal threshold of 13 pieces. Dec 26 17:00:44.056062314 (505h31m43.927145349s) [noise=true , address=45.91.116.14:4189 , piece_id=M4FYM3IRUT3N6BBCU7Z2RYC4YTEZ7QV3T5733BZU5SAKGQMDBP6Q, seg_pos={0 0}, upload=443596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:45810->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:45810->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45810->45.91.116.14:4189: use of closed network connection Dec 26 17:00:44.056069407 (505h31m43.927152443s) [noise=true , address=45.91.116.14:4189 , piece_id=M4FYM3IRUT3N6BBCU7Z2RYC4YTEZ7QV3T5733BZU5SAKGQMDBP6Q, seg_pos={0 0}, upload=443596]: Long tail cancellation due to optimal threshold Dec 26 17:00:44.056471353 (505h31m43.927554398s) [noise=true , address=45.91.116.6:4974 , piece_id=5P2OWJ74S3DDQ3CGYFL2KIAZUU3YO5TZMKQ2ZCBI24URYLR45REA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 26 17:00:44.056482484 (505h31m43.927565519s) [noise=true , address=45.91.116.6:4974 , piece_id=5P2OWJ74S3DDQ3CGYFL2KIAZUU3YO5TZMKQ2ZCBI24URYLR45REA, seg_pos={0 0}, upload=443596]: Long tail cancellation due to optimal threshold Dec 26 17:00:44.056502382 (505h31m43.927585418s) [seg_pos={0 0}, upload=443596]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:00:44.056585761 (505h31m43.927668806s) [upload=443596]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:00:44.056592714 (505h31m43.927675749s) [upload=443596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:00:44.356920100 (505h31m44.228003135s) [upload=443596]: Done uploading segments... Dec 26 17:00:44.356939787 (505h31m44.228022822s) [upload=443596]: Upload finished. err: <nil> Duration: 6.153149862s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.888430281s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:01:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 6.888450148s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.XX.....XX..........X......XXX...X........X.................] duration: 3.486252458s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:01:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.486280492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX.X.XX.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX] duration: 30.004331665s trace: https://jaeger.dev.storj.io/trace/2ea6b7cc03d87efc FAIL (2025-12-26 17:01:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:00:41.074546763 (505h31m40.945629798s) [upload=443596]: Starting upload Dec 26 17:00:41.074674687 (505h31m40.945757732s) [upload=443596]: Done starting upload Dec 26 17:00:41.074739359 (505h31m40.945822404s) [upload=443596]: Uploading segments... Dec 26 17:00:41.074970079 (505h31m40.946053114s) [upload=443596]: Got next segment. Inline: false Dec 26 17:00:41.075003883 (505h31m40.946086929s) [upload=443596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:00:41.075008482 (505h31m40.946091517s) [upload=443596]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:00:41.322478708 (505h31m41.193561763s) [seg_pos={0 0}, upload=443596]: Begin upload segment... Dec 26 17:00:41.322545916 (505h31m41.193628961s) [seg_pos={0 0}, upload=443596]: Done begin upload segment. Dec 26 17:00:41.322605950 (505h31m41.193688985s) [noise=true , address=45.91.116.10:4084 , piece_id=L2IDPQS227IRJPUFCIPZICAJUNLZ2TGV7KICBAHBTG3JN66DKRNQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.322650866 (505h31m41.193733911s) [noise=true , address=145.79.227.17:4971 , piece_id=S2PMFG3NELSF7SFEIQTCYW4RY6AP62JET4JKAD5K4X6UP7I5WJKQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.323465316 (505h31m41.194548361s) [noise=true , address=77.90.31.11:4555 , piece_id=CKUTE7UJUE6NCN3TCX4MZVT3HWQTGEDVR2QQIFW4DDFVH7SIGA4A, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.324090377 (505h31m41.195173412s) [noise=true , address=145.79.227.39:4015 , piece_id=JRNT3EMKFDNR7NSKKBEZ4W73CTEQDHBDI2AN4SDC6Z5OLVEFVDZA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.324456204 (505h31m41.195539249s) [noise=true , address=145.79.227.41:4087 , piece_id=INHEWTEIW2YLVUJZ3RJE75SQZPGAM4LTRQKA4CYVOSK5XM7HSDRA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.324950565 (505h31m41.1960336s) [noise=true , address=145.79.227.22:4511 , piece_id=TNYHQSYXHHIRGGO3WU5ISZD4CKSEH6FZJITA7WO6PQGA2EDCY3WQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.325394541 (505h31m41.196477576s) [noise=true , address=45.91.116.6:4974 , piece_id=5P2OWJ74S3DDQ3CGYFL2KIAZUU3YO5TZMKQ2ZCBI24URYLR45REA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.325400442 (505h31m41.196483487s) [noise=true , address=145.79.227.45:4230 , piece_id=NCGLLCQ6G35KHW7RTR6RMWBCFGE643QMPWGPHMNLRRV2XO4AKJXA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.326139929 (505h31m41.197222975s) [noise=true , address=145.79.227.15:4876 , piece_id=MF63IBUO3K3OJJE53JVTUPGND6OA5SP7OSUGVVQ2TM3LNGTTBSMQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.326761233 (505h31m41.197844278s) [noise=true , address=145.79.227.52:4445 , piece_id=JNR2GDK4ZL32GH3L5QILVLFQMGNEHU4DC6X6DCFL5YH7IDKN5OZA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.327559563 (505h31m41.198642608s) [noise=true , address=45.91.116.13:4154 , piece_id=LBVYTFLJM6XVNXPPRBZX3577V5YG4AXXKW4UFAKBD32SEV326XDA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.328178341 (505h31m41.199261386s) [noise=true , address=145.79.227.37:4566 , piece_id=HJFAOPCO7BK3K6FIJHJGH6Y5HBTFXGFRCHW7BSAWYIUS2QMXZPQA, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.330384672 (505h31m41.201467717s) [noise=true , address=145.79.227.2:4113 , piece_id=S3AUHP2P5LE4JIFSSXXA7LM4YB5VFHYNGZBXYTIEYDFNDS2ZI5IQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.332356868 (505h31m41.203439913s) [noise=true , address=45.91.116.14:4189 , piece_id=M4FYM3IRUT3N6BBCU7Z2RYC4YTEZ7QV3T5733BZU5SAKGQMDBP6Q, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.334755154 (505h31m41.205838199s) [noise=true , address=145.79.227.47:4266 , piece_id=5GOIJJ5KB2ED72P7CJ3CJKJRDI76D5NVWZ4BC2NAJZ5BJ4KT2ZTQ, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:41.336744082 (505h31m41.207827117s) [noise=true , address=145.79.227.11:4731 , piece_id=3UD2662V7S5FW2JAFAHWDWFHNZ3J6II4HLKMKBG6XIOYBTAUR33Q, seg_pos={0 0}, upload=443596]: Uploading piece... Dec 26 17:00:42.487244964 (505h31m42.358328019s) [noise=true , address=77.90.31.11:4555 , piece_id=CKUTE7UJUE6NCN3TCX4MZVT3HWQTGEDVR2QQIFW4DDFVH7SIGA4A, seg_pos={0 0}, upload=443596]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: storage node overloaded, request limit: 2000 Dec 26 17:00:43.031756910 (505h31m42.902839945s) [upload=443596]: Next returned nil segment Dec 26 17:00:43.031782708 (505h31m42.902865753s) [upload=443596]: Waiting for error group managing segments... Dec 26 17:00:43.316897206 (505h31m43.187980251s) [noise=true , address=145.79.227.17:4971 , piece_id=S2PMFG3NELSF7SFEIQTCYW4RY6AP62JET4JKAD5K4X6UP7I5WJKQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.332085437 (505h31m43.203168492s) [noise=true , address=145.79.227.39:4015 , piece_id=JRNT3EMKFDNR7NSKKBEZ4W73CTEQDHBDI2AN4SDC6Z5OLVEFVDZA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.333057478 (505h31m43.204140513s) [noise=true , address=145.79.227.41:4087 , piece_id=INHEWTEIW2YLVUJZ3RJE75SQZPGAM4LTRQKA4CYVOSK5XM7HSDRA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.341314691 (505h31m43.212397726s) [noise=true , address=145.79.227.15:4876 , piece_id=MF63IBUO3K3OJJE53JVTUPGND6OA5SP7OSUGVVQ2TM3LNGTTBSMQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.343019046 (505h31m43.214102081s) [noise=true , address=145.79.227.45:4230 , piece_id=NCGLLCQ6G35KHW7RTR6RMWBCFGE643QMPWGPHMNLRRV2XO4AKJXA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.344413391 (505h31m43.215496436s) [noise=true , address=145.79.227.37:4566 , piece_id=HJFAOPCO7BK3K6FIJHJGH6Y5HBTFXGFRCHW7BSAWYIUS2QMXZPQA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.344517990 (505h31m43.215601025s) [noise=true , address=145.79.227.11:4731 , piece_id=3UD2662V7S5FW2JAFAHWDWFHNZ3J6II4HLKMKBG6XIOYBTAUR33Q, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.350207134 (505h31m43.221290179s) [noise=true , address=145.79.227.22:4511 , piece_id=TNYHQSYXHHIRGGO3WU5ISZD4CKSEH6FZJITA7WO6PQGA2EDCY3WQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.367585716 (505h31m43.238668751s) [noise=true , address=145.79.227.52:4445 , piece_id=JNR2GDK4ZL32GH3L5QILVLFQMGNEHU4DC6X6DCFL5YH7IDKN5OZA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.379997721 (505h31m43.251080756s) [noise=true , address=145.79.227.47:4266 , piece_id=5GOIJJ5KB2ED72P7CJ3CJKJRDI76D5NVWZ4BC2NAJZ5BJ4KT2ZTQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.388167787 (505h31m43.259250832s) [noise=true , address=145.79.227.2:4113 , piece_id=S3AUHP2P5LE4JIFSSXXA7LM4YB5VFHYNGZBXYTIEYDFNDS2ZI5IQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:43.698245359 (505h31m43.569328404s) [noise=true , address=45.91.116.10:4084 , piece_id=L2IDPQS227IRJPUFCIPZICAJUNLZ2TGV7KICBAHBTG3JN66DKRNQ, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:44.055826906 (505h31m43.926909941s) [noise=true , address=45.91.116.13:4154 , piece_id=LBVYTFLJM6XVNXPPRBZX3577V5YG4AXXKW4UFAKBD32SEV326XDA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: <nil> Dec 26 17:00:44.055845892 (505h31m43.926928927s) [seg_pos={0 0}, upload=443596]: Segment reached optimal threshold of 13 pieces. Dec 26 17:00:44.056062314 (505h31m43.927145349s) [noise=true , address=45.91.116.14:4189 , piece_id=M4FYM3IRUT3N6BBCU7Z2RYC4YTEZ7QV3T5733BZU5SAKGQMDBP6Q, seg_pos={0 0}, upload=443596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:45810->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:45810->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45810->45.91.116.14:4189: use of closed network connection Dec 26 17:00:44.056069407 (505h31m43.927152443s) [noise=true , address=45.91.116.14:4189 , piece_id=M4FYM3IRUT3N6BBCU7Z2RYC4YTEZ7QV3T5733BZU5SAKGQMDBP6Q, seg_pos={0 0}, upload=443596]: Long tail cancellation due to optimal threshold Dec 26 17:00:44.056471353 (505h31m43.927554398s) [noise=true , address=45.91.116.6:4974 , piece_id=5P2OWJ74S3DDQ3CGYFL2KIAZUU3YO5TZMKQ2ZCBI24URYLR45REA, seg_pos={0 0}, upload=443596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 26 17:00:44.056482484 (505h31m43.927565519s) [noise=true , address=45.91.116.6:4974 , piece_id=5P2OWJ74S3DDQ3CGYFL2KIAZUU3YO5TZMKQ2ZCBI24URYLR45REA, seg_pos={0 0}, upload=443596]: Long tail cancellation due to optimal threshold Dec 26 17:00:44.056502382 (505h31m43.927585418s) [seg_pos={0 0}, upload=443596]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:00:44.056585761 (505h31m43.927668806s) [upload=443596]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:00:44.056592714 (505h31m43.927675749s) [upload=443596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:00:44.356920100 (505h31m44.228003135s) [upload=443596]: Done uploading segments... Dec 26 17:00:44.356939787 (505h31m44.228022822s) [upload=443596]: Upload finished. err: <nil> Duration: 30.0043448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.........X..X...............X.......X....X................] duration: 47.459884388s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:00:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 47.459914484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.454425081s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:00:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 6.454454237s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)