Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.........................................X...........X...] duration: 15.410768626s trace: https://jaeger.dev.storj.io/trace/1a677c844092c76b FAIL (2025-12-31 01:17:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.800832805 (609h45m28.67191585s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801258315 (609h45m28.67234135s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801566793 (609h45m28.672649828s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801962346 (609h45m28.673045391s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.802631961 (609h45m28.673715006s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.803220893 (609h45m28.674303928s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804321939 (609h45m28.675404984s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804615839 (609h45m28.675698874s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804969001 (609h45m28.676052036s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805292477 (609h45m28.676375512s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805617435 (609h45m28.676700471s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806029400 (609h45m28.677112445s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806610056 (609h45m28.677693111s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:30.220457684 (609h45m30.091540719s) [upload=534376]: Next returned nil segment Dec 31 01:14:30.220473895 (609h45m30.09155693s) [upload=534376]: Waiting for error group managing segments... Dec 31 01:14:30.582370671 (609h45m30.453453706s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.660359852 (609h45m30.531442897s) [noise=true , address=145.79.227.29:4961 , piece_id=FL4ZAH3X7BJZZJZ2VLAOXZSS7DZ5NGUYGEYX2II7J6P6ERRKMWSA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669673326 (609h45m30.540756361s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669821237 (609h45m30.540904272s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.723128962 (609h45m30.594212007s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.836497239 (609h45m30.707580284s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.847254552 (609h45m30.718337587s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.857173689 (609h45m30.728256734s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.956827463 (609h45m30.827910498s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.987365921 (609h45m30.858448956s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.032233144 (609h45m30.903316179s) [noise=true , address=45.91.116.8:4055 , piece_id=SC3SJFOKLLANT6EZC42UWON7YW73HGZAWICGMV7EY2CGS5VO3R4A, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.549259104 (609h45m31.420342149s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.563439376 (609h45m31.434522421s) [noise=true , address=145.79.227.43:4171 , piece_id=NUT3FLC5ORYYHMESA5MRPQSYKSLVBMNDURGSW4LG5MW4WHZZCHWA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613378095 (609h45m31.48446114s) [noise=true , address=145.79.227.44:4204 , piece_id=MSLSLTQFLQDKLOUDSD3BVSTD54NGEDE72EHWHYZCWT7AX3GZF2VQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613397552 (609h45m31.484480597s) [seg_pos={0 0}, upload=534376]: Segment reached optimal threshold of 14 pieces. Dec 31 01:14:31.613581703 (609h45m31.484664738s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12238rvPqrDXBowLsUiKTykdZFgfyq4fX9YdaeLa52sQJCkRMHM): protocol: expected piece hash; context canceled; EOF Dec 31 01:14:31.613588355 (609h45m31.48467139s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613657807 (609h45m31.484740853s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta): protocol: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection Dec 31 01:14:31.613666333 (609h45m31.484749369s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613690650 (609h45m31.484773695s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection Dec 31 01:14:31.613696842 (609h45m31.484779887s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613847469 (609h45m31.484930514s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgazvjebAo7K4NdQ7zsKkETiNQNfiykHcBDcmXej6eipwWpFu): protocol: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection Dec 31 01:14:31.613854532 (609h45m31.484937577s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613947559 (609h45m31.485030594s) [seg_pos={0 0}, upload=534376]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:14:31.614017652 (609h45m31.485100697s) [upload=534376]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:14:31.614023655 (609h45m31.48510669s) [upload=534376]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:14:31.917383268 (609h45m31.788466313s) [upload=534376]: Done uploading segments... Dec 31 01:14:31.917408336 (609h45m31.788491381s) [upload=534376]: Upload finished. err: <nil> Duration: 15.410797831s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.61456654s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:17:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.614595424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.51715491s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:16:47 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.517184966s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....X...] duration: 23.952296968s trace: https://jaeger.dev.storj.io/trace/5c211acadf0a2f7c FAIL (2025-12-31 01:16:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:13:31.892162068 (609h44m31.763245103s) [upload=534365]: Starting upload Dec 31 01:13:31.892229988 (609h44m31.763313023s) [upload=534365]: Done starting upload Dec 31 01:13:31.892520051 (609h44m31.763603086s) [upload=534365]: Uploading segments... Dec 31 01:13:31.892835572 (609h44m31.763918607s) [upload=534365]: Got next segment. Inline: false Dec 31 01:13:31.892880577 (609h44m31.763963622s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:13:31.892886969 (609h44m31.763970014s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:13:32.136582035 (609h44m32.00766508s) [seg_pos={0 0}, upload=534365]: Begin upload segment... Dec 31 01:13:32.136741648 (609h44m32.007824693s) [seg_pos={0 0}, upload=534365]: Done begin upload segment. Dec 31 01:13:32.136787886 (609h44m32.007870921s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.136952009 (609h44m32.008035054s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137392508 (609h44m32.008475553s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137833196 (609h44m32.008916241s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137887390 (609h44m32.008970435s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138202520 (609h44m32.009285555s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138443369 (609h44m32.009526414s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138677505 (609h44m32.00976054s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138997634 (609h44m32.010080679s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139179520 (609h44m32.010262565s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139293667 (609h44m32.010376712s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139647271 (609h44m32.010730316s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139782598 (609h44m32.010865643s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.140427918 (609h44m32.011510953s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141112661 (609h44m32.012195706s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141783057 (609h44m32.012866102s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:33.837053250 (609h44m33.708136285s) [upload=534365]: Next returned nil segment Dec 31 01:13:33.837088717 (609h44m33.708171752s) [upload=534365]: Waiting for error group managing segments... Dec 31 01:13:34.123562999 (609h44m33.994646034s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.124260658 (609h44m33.995343693s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.125012440 (609h44m33.996095485s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.132053355 (609h44m34.003136411s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.135253870 (609h44m34.006336905s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.138185082 (609h44m34.009268117s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.145248080 (609h44m34.016331116s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.146704714 (609h44m34.017787779s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.312085941 (609h44m34.183168976s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.436213054 (609h44m34.307296109s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.766896134 (609h44m34.637979179s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.034120011 (609h44m34.905203046s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317152980 (609h44m35.188236025s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317171395 (609h44m35.18825443s) [seg_pos={0 0}, upload=534365]: Segment reached optimal threshold of 13 pieces. Dec 31 01:13:35.317363992 (609h44m35.188447027s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317369673 (609h44m35.188452708s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317497957 (609h44m35.188580992s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317503487 (609h44m35.188586522s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317612284 (609h44m35.188695339s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317643834 (609h44m35.188726869s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317682628 (609h44m35.188765663s) [seg_pos={0 0}, upload=534365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:13:35.317778912 (609h44m35.188861947s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:13:35.317784572 (609h44m35.188867607s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:13:35.615134704 (609h44m35.486217749s) [upload=534365]: Done uploading segments... Dec 31 01:13:35.615159452 (609h44m35.486242497s) [upload=534365]: Upload finished. err: <nil> Duration: 23.952325311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.........................................X...........X..] duration: 17.437483064s trace: https://jaeger.dev.storj.io/trace/1a677c844092c76b FAIL (2025-12-31 01:16:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.800832805 (609h45m28.67191585s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801258315 (609h45m28.67234135s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801566793 (609h45m28.672649828s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801962346 (609h45m28.673045391s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.802631961 (609h45m28.673715006s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.803220893 (609h45m28.674303928s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804321939 (609h45m28.675404984s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804615839 (609h45m28.675698874s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804969001 (609h45m28.676052036s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805292477 (609h45m28.676375512s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805617435 (609h45m28.676700471s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806029400 (609h45m28.677112445s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806610056 (609h45m28.677693111s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:30.220457684 (609h45m30.091540719s) [upload=534376]: Next returned nil segment Dec 31 01:14:30.220473895 (609h45m30.09155693s) [upload=534376]: Waiting for error group managing segments... Dec 31 01:14:30.582370671 (609h45m30.453453706s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.660359852 (609h45m30.531442897s) [noise=true , address=145.79.227.29:4961 , piece_id=FL4ZAH3X7BJZZJZ2VLAOXZSS7DZ5NGUYGEYX2II7J6P6ERRKMWSA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669673326 (609h45m30.540756361s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669821237 (609h45m30.540904272s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.723128962 (609h45m30.594212007s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.836497239 (609h45m30.707580284s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.847254552 (609h45m30.718337587s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.857173689 (609h45m30.728256734s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.956827463 (609h45m30.827910498s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.987365921 (609h45m30.858448956s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.032233144 (609h45m30.903316179s) [noise=true , address=45.91.116.8:4055 , piece_id=SC3SJFOKLLANT6EZC42UWON7YW73HGZAWICGMV7EY2CGS5VO3R4A, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.549259104 (609h45m31.420342149s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.563439376 (609h45m31.434522421s) [noise=true , address=145.79.227.43:4171 , piece_id=NUT3FLC5ORYYHMESA5MRPQSYKSLVBMNDURGSW4LG5MW4WHZZCHWA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613378095 (609h45m31.48446114s) [noise=true , address=145.79.227.44:4204 , piece_id=MSLSLTQFLQDKLOUDSD3BVSTD54NGEDE72EHWHYZCWT7AX3GZF2VQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613397552 (609h45m31.484480597s) [seg_pos={0 0}, upload=534376]: Segment reached optimal threshold of 14 pieces. Dec 31 01:14:31.613581703 (609h45m31.484664738s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12238rvPqrDXBowLsUiKTykdZFgfyq4fX9YdaeLa52sQJCkRMHM): protocol: expected piece hash; context canceled; EOF Dec 31 01:14:31.613588355 (609h45m31.48467139s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613657807 (609h45m31.484740853s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta): protocol: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection Dec 31 01:14:31.613666333 (609h45m31.484749369s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613690650 (609h45m31.484773695s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection Dec 31 01:14:31.613696842 (609h45m31.484779887s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613847469 (609h45m31.484930514s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgazvjebAo7K4NdQ7zsKkETiNQNfiykHcBDcmXej6eipwWpFu): protocol: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection Dec 31 01:14:31.613854532 (609h45m31.484937577s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613947559 (609h45m31.485030594s) [seg_pos={0 0}, upload=534376]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:14:31.614017652 (609h45m31.485100697s) [upload=534376]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:14:31.614023655 (609h45m31.48510669s) [upload=534376]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:14:31.917383268 (609h45m31.788466313s) [upload=534376]: Done uploading segments... Dec 31 01:14:31.917408336 (609h45m31.788491381s) [upload=534376]: Upload finished. err: <nil> Duration: 17.437511157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.504170178s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:16:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.504196378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.309370451s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:16:07 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.309403764s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.40283603s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:15:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.402915681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.........................................X...........X.] duration: 17.548250581s trace: https://jaeger.dev.storj.io/trace/1a677c844092c76b FAIL (2025-12-31 01:15:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.800832805 (609h45m28.67191585s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801258315 (609h45m28.67234135s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801566793 (609h45m28.672649828s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801962346 (609h45m28.673045391s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.802631961 (609h45m28.673715006s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.803220893 (609h45m28.674303928s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804321939 (609h45m28.675404984s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804615839 (609h45m28.675698874s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804969001 (609h45m28.676052036s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805292477 (609h45m28.676375512s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805617435 (609h45m28.676700471s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806029400 (609h45m28.677112445s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806610056 (609h45m28.677693111s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:30.220457684 (609h45m30.091540719s) [upload=534376]: Next returned nil segment Dec 31 01:14:30.220473895 (609h45m30.09155693s) [upload=534376]: Waiting for error group managing segments... Dec 31 01:14:30.582370671 (609h45m30.453453706s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.660359852 (609h45m30.531442897s) [noise=true , address=145.79.227.29:4961 , piece_id=FL4ZAH3X7BJZZJZ2VLAOXZSS7DZ5NGUYGEYX2II7J6P6ERRKMWSA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669673326 (609h45m30.540756361s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669821237 (609h45m30.540904272s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.723128962 (609h45m30.594212007s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.836497239 (609h45m30.707580284s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.847254552 (609h45m30.718337587s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.857173689 (609h45m30.728256734s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.956827463 (609h45m30.827910498s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.987365921 (609h45m30.858448956s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.032233144 (609h45m30.903316179s) [noise=true , address=45.91.116.8:4055 , piece_id=SC3SJFOKLLANT6EZC42UWON7YW73HGZAWICGMV7EY2CGS5VO3R4A, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.549259104 (609h45m31.420342149s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.563439376 (609h45m31.434522421s) [noise=true , address=145.79.227.43:4171 , piece_id=NUT3FLC5ORYYHMESA5MRPQSYKSLVBMNDURGSW4LG5MW4WHZZCHWA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613378095 (609h45m31.48446114s) [noise=true , address=145.79.227.44:4204 , piece_id=MSLSLTQFLQDKLOUDSD3BVSTD54NGEDE72EHWHYZCWT7AX3GZF2VQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613397552 (609h45m31.484480597s) [seg_pos={0 0}, upload=534376]: Segment reached optimal threshold of 14 pieces. Dec 31 01:14:31.613581703 (609h45m31.484664738s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12238rvPqrDXBowLsUiKTykdZFgfyq4fX9YdaeLa52sQJCkRMHM): protocol: expected piece hash; context canceled; EOF Dec 31 01:14:31.613588355 (609h45m31.48467139s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613657807 (609h45m31.484740853s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta): protocol: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection Dec 31 01:14:31.613666333 (609h45m31.484749369s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613690650 (609h45m31.484773695s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection Dec 31 01:14:31.613696842 (609h45m31.484779887s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613847469 (609h45m31.484930514s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgazvjebAo7K4NdQ7zsKkETiNQNfiykHcBDcmXej6eipwWpFu): protocol: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection Dec 31 01:14:31.613854532 (609h45m31.484937577s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613947559 (609h45m31.485030594s) [seg_pos={0 0}, upload=534376]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:14:31.614017652 (609h45m31.485100697s) [upload=534376]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:14:31.614023655 (609h45m31.48510669s) [upload=534376]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:14:31.917383268 (609h45m31.788466313s) [upload=534376]: Done uploading segments... Dec 31 01:14:31.917408336 (609h45m31.788491381s) [upload=534376]: Upload finished. err: <nil> Duration: 17.548263527s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.574912883s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:15:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.574958029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.........................................X...........X] duration: 1m0.003237406s trace: https://jaeger.dev.storj.io/trace/1a677c844092c76b FAIL (2025-12-31 01:15:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.800832805 (609h45m28.67191585s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801258315 (609h45m28.67234135s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801566793 (609h45m28.672649828s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.801962346 (609h45m28.673045391s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.802631961 (609h45m28.673715006s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.803220893 (609h45m28.674303928s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804321939 (609h45m28.675404984s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804615839 (609h45m28.675698874s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.804969001 (609h45m28.676052036s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805292477 (609h45m28.676375512s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.805617435 (609h45m28.676700471s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806029400 (609h45m28.677112445s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:28.806610056 (609h45m28.677693111s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Uploading piece... Dec 31 01:14:30.220457684 (609h45m30.091540719s) [upload=534376]: Next returned nil segment Dec 31 01:14:30.220473895 (609h45m30.09155693s) [upload=534376]: Waiting for error group managing segments... Dec 31 01:14:30.582370671 (609h45m30.453453706s) [noise=true , address=64.49.13.170:4727 , piece_id=4RMUNBIBYA7HZ4Y6XQSSHY4Z6Q26RUA3OBJXKWLZ5X53B5UJHFUA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.660359852 (609h45m30.531442897s) [noise=true , address=145.79.227.29:4961 , piece_id=FL4ZAH3X7BJZZJZ2VLAOXZSS7DZ5NGUYGEYX2II7J6P6ERRKMWSA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669673326 (609h45m30.540756361s) [noise=true , address=145.79.227.6:4280 , piece_id=DVS4HVIFEX6HY3NPBQC5J2PTTDCQ3Y7LRAHA76FAQKGMMVRK5RLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.669821237 (609h45m30.540904272s) [noise=true , address=145.79.227.16:4918 , piece_id=KTNWZCL4OHO34WOREGQ5LBMANTRX5NKS6SMPFJUV7JQSTKSTQYDA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.723128962 (609h45m30.594212007s) [noise=true , address=145.79.227.9:4375 , piece_id=DBIJYHETD5TOBQQBQBQUYXWYENNTPCCTTR6NKYWO36XDPDIHJGRQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.836497239 (609h45m30.707580284s) [noise=true , address=77.90.31.14:4684 , piece_id=UOGNPGQMJ7S5LSQDC7KKKBQ2MEZRXGFGWHLAUC54R6PKRGQ3MDVQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.847254552 (609h45m30.718337587s) [noise=true , address=77.90.31.12:4617 , piece_id=5HOAYWDAAIPPGAEMZ2RGYD33ADMQK2ARK6VA7G77HTAGP3SEDXZQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.857173689 (609h45m30.728256734s) [noise=true , address=77.90.31.10:4526 , piece_id=35AWY74SYNHQAXJXRIMWXRENPCA7C2KEM3IZV5H7RK2R3REV4NVA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.956827463 (609h45m30.827910498s) [noise=true , address=45.91.116.7:4027 , piece_id=BAJMVHMEBVKC753XA6ZBO7YVFKR4IOTVGKTJIUSLKFXDTJENRMEQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:30.987365921 (609h45m30.858448956s) [noise=true , address=145.79.227.28:4743 , piece_id=4I4EL63MIAAIUV3WS2SKF6H2EM3II4TL5SEXDZHSD2HQR35RAZUQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.032233144 (609h45m30.903316179s) [noise=true , address=45.91.116.8:4055 , piece_id=SC3SJFOKLLANT6EZC42UWON7YW73HGZAWICGMV7EY2CGS5VO3R4A, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.549259104 (609h45m31.420342149s) [noise=true , address=145.79.227.40:4059 , piece_id=MH7WUTLHFU7PH5CPP6EZFQEFLVGLG4TZP2PZROUA4A5VAQJWBHIQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.563439376 (609h45m31.434522421s) [noise=true , address=145.79.227.43:4171 , piece_id=NUT3FLC5ORYYHMESA5MRPQSYKSLVBMNDURGSW4LG5MW4WHZZCHWA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613378095 (609h45m31.48446114s) [noise=true , address=145.79.227.44:4204 , piece_id=MSLSLTQFLQDKLOUDSD3BVSTD54NGEDE72EHWHYZCWT7AX3GZF2VQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: <nil> Dec 31 01:14:31.613397552 (609h45m31.484480597s) [seg_pos={0 0}, upload=534376]: Segment reached optimal threshold of 14 pieces. Dec 31 01:14:31.613581703 (609h45m31.484664738s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12238rvPqrDXBowLsUiKTykdZFgfyq4fX9YdaeLa52sQJCkRMHM): protocol: expected piece hash; context canceled; EOF Dec 31 01:14:31.613588355 (609h45m31.48467139s) [noise=true , address=145.79.227.49:4362 , piece_id=DC3BUO6NTIGJ46L6UKQFLOP2HVVNMWA2YE2VILGHW4K2ES4GWVAA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613657807 (609h45m31.484740853s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta): protocol: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54740->145.79.227.18:4031: use of closed network connection Dec 31 01:14:31.613666333 (609h45m31.484749369s) [noise=true , address=145.79.227.18:4031 , piece_id=32G6HXKB6Y5SDM4GGILLUKLDVXEHIE3SRPEDFR7XF3KSUIWDLBCQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613690650 (609h45m31.484773695s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34536->145.79.227.2:4141: use of closed network connection Dec 31 01:14:31.613696842 (609h45m31.484779887s) [noise=true , address=145.79.227.2:4141 , piece_id=AEITGTITF4IJNFTDL6P6B3UI3QEZMHFPK36WO7YAECB5BB3XEQLA, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613847469 (609h45m31.484930514s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgazvjebAo7K4NdQ7zsKkETiNQNfiykHcBDcmXej6eipwWpFu): protocol: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48222->145.79.227.4:4216: use of closed network connection Dec 31 01:14:31.613854532 (609h45m31.484937577s) [noise=true , address=145.79.227.4:4216 , piece_id=MYGW7QWRK2UF4J4J3LGYZCKFITF7YWYH24TCZD7CP2LD2S5UQTQQ, seg_pos={0 0}, upload=534376]: Long tail cancellation due to optimal threshold Dec 31 01:14:31.613947559 (609h45m31.485030594s) [seg_pos={0 0}, upload=534376]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:14:31.614017652 (609h45m31.485100697s) [upload=534376]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:14:31.614023655 (609h45m31.48510669s) [upload=534376]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:14:31.917383268 (609h45m31.788466313s) [upload=534376]: Done uploading segments... Dec 31 01:14:31.917408336 (609h45m31.788491381s) [upload=534376]: Upload finished. err: <nil> Duration: 1m0.003251011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....X..] duration: 5.81946758s trace: https://jaeger.dev.storj.io/trace/5c211acadf0a2f7c FAIL (2025-12-31 01:15:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:13:31.892162068 (609h44m31.763245103s) [upload=534365]: Starting upload Dec 31 01:13:31.892229988 (609h44m31.763313023s) [upload=534365]: Done starting upload Dec 31 01:13:31.892520051 (609h44m31.763603086s) [upload=534365]: Uploading segments... Dec 31 01:13:31.892835572 (609h44m31.763918607s) [upload=534365]: Got next segment. Inline: false Dec 31 01:13:31.892880577 (609h44m31.763963622s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:13:31.892886969 (609h44m31.763970014s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:13:32.136582035 (609h44m32.00766508s) [seg_pos={0 0}, upload=534365]: Begin upload segment... Dec 31 01:13:32.136741648 (609h44m32.007824693s) [seg_pos={0 0}, upload=534365]: Done begin upload segment. Dec 31 01:13:32.136787886 (609h44m32.007870921s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.136952009 (609h44m32.008035054s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137392508 (609h44m32.008475553s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137833196 (609h44m32.008916241s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137887390 (609h44m32.008970435s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138202520 (609h44m32.009285555s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138443369 (609h44m32.009526414s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138677505 (609h44m32.00976054s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138997634 (609h44m32.010080679s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139179520 (609h44m32.010262565s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139293667 (609h44m32.010376712s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139647271 (609h44m32.010730316s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139782598 (609h44m32.010865643s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.140427918 (609h44m32.011510953s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141112661 (609h44m32.012195706s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141783057 (609h44m32.012866102s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:33.837053250 (609h44m33.708136285s) [upload=534365]: Next returned nil segment Dec 31 01:13:33.837088717 (609h44m33.708171752s) [upload=534365]: Waiting for error group managing segments... Dec 31 01:13:34.123562999 (609h44m33.994646034s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.124260658 (609h44m33.995343693s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.125012440 (609h44m33.996095485s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.132053355 (609h44m34.003136411s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.135253870 (609h44m34.006336905s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.138185082 (609h44m34.009268117s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.145248080 (609h44m34.016331116s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.146704714 (609h44m34.017787779s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.312085941 (609h44m34.183168976s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.436213054 (609h44m34.307296109s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.766896134 (609h44m34.637979179s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.034120011 (609h44m34.905203046s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317152980 (609h44m35.188236025s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317171395 (609h44m35.18825443s) [seg_pos={0 0}, upload=534365]: Segment reached optimal threshold of 13 pieces. Dec 31 01:13:35.317363992 (609h44m35.188447027s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317369673 (609h44m35.188452708s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317497957 (609h44m35.188580992s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317503487 (609h44m35.188586522s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317612284 (609h44m35.188695339s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317643834 (609h44m35.188726869s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317682628 (609h44m35.188765663s) [seg_pos={0 0}, upload=534365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:13:35.317778912 (609h44m35.188861947s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:13:35.317784572 (609h44m35.188867607s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:13:35.615134704 (609h44m35.486217749s) [upload=534365]: Done uploading segments... Dec 31 01:13:35.615159452 (609h44m35.486242497s) [upload=534365]: Upload finished. err: <nil> Duration: 5.819497608s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.06753071s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:15:03 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.06756245s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.534757902s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:14:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534783531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.........................................X...........] duration: 45.685957022s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:14:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 45.685996447s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.563277108s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:14:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563305953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.040206151s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:14:14 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.040233513s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....X.] duration: 5.955976597s trace: https://jaeger.dev.storj.io/trace/5c211acadf0a2f7c FAIL (2025-12-31 01:14:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:13:31.892162068 (609h44m31.763245103s) [upload=534365]: Starting upload Dec 31 01:13:31.892229988 (609h44m31.763313023s) [upload=534365]: Done starting upload Dec 31 01:13:31.892520051 (609h44m31.763603086s) [upload=534365]: Uploading segments... Dec 31 01:13:31.892835572 (609h44m31.763918607s) [upload=534365]: Got next segment. Inline: false Dec 31 01:13:31.892880577 (609h44m31.763963622s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:13:31.892886969 (609h44m31.763970014s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:13:32.136582035 (609h44m32.00766508s) [seg_pos={0 0}, upload=534365]: Begin upload segment... Dec 31 01:13:32.136741648 (609h44m32.007824693s) [seg_pos={0 0}, upload=534365]: Done begin upload segment. Dec 31 01:13:32.136787886 (609h44m32.007870921s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.136952009 (609h44m32.008035054s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137392508 (609h44m32.008475553s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137833196 (609h44m32.008916241s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137887390 (609h44m32.008970435s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138202520 (609h44m32.009285555s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138443369 (609h44m32.009526414s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138677505 (609h44m32.00976054s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138997634 (609h44m32.010080679s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139179520 (609h44m32.010262565s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139293667 (609h44m32.010376712s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139647271 (609h44m32.010730316s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139782598 (609h44m32.010865643s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.140427918 (609h44m32.011510953s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141112661 (609h44m32.012195706s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141783057 (609h44m32.012866102s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:33.837053250 (609h44m33.708136285s) [upload=534365]: Next returned nil segment Dec 31 01:13:33.837088717 (609h44m33.708171752s) [upload=534365]: Waiting for error group managing segments... Dec 31 01:13:34.123562999 (609h44m33.994646034s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.124260658 (609h44m33.995343693s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.125012440 (609h44m33.996095485s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.132053355 (609h44m34.003136411s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.135253870 (609h44m34.006336905s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.138185082 (609h44m34.009268117s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.145248080 (609h44m34.016331116s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.146704714 (609h44m34.017787779s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.312085941 (609h44m34.183168976s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.436213054 (609h44m34.307296109s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.766896134 (609h44m34.637979179s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.034120011 (609h44m34.905203046s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317152980 (609h44m35.188236025s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317171395 (609h44m35.18825443s) [seg_pos={0 0}, upload=534365]: Segment reached optimal threshold of 13 pieces. Dec 31 01:13:35.317363992 (609h44m35.188447027s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317369673 (609h44m35.188452708s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317497957 (609h44m35.188580992s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317503487 (609h44m35.188586522s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317612284 (609h44m35.188695339s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317643834 (609h44m35.188726869s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317682628 (609h44m35.188765663s) [seg_pos={0 0}, upload=534365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:13:35.317778912 (609h44m35.188861947s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:13:35.317784572 (609h44m35.188867607s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:13:35.615134704 (609h44m35.486217749s) [upload=534365]: Done uploading segments... Dec 31 01:13:35.615159452 (609h44m35.486242497s) [upload=534365]: Upload finished. err: <nil> Duration: 5.956006524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....X] duration: 30.00279558s trace: https://jaeger.dev.storj.io/trace/5c211acadf0a2f7c FAIL (2025-12-31 01:14:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:13:31.892162068 (609h44m31.763245103s) [upload=534365]: Starting upload Dec 31 01:13:31.892229988 (609h44m31.763313023s) [upload=534365]: Done starting upload Dec 31 01:13:31.892520051 (609h44m31.763603086s) [upload=534365]: Uploading segments... Dec 31 01:13:31.892835572 (609h44m31.763918607s) [upload=534365]: Got next segment. Inline: false Dec 31 01:13:31.892880577 (609h44m31.763963622s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:13:31.892886969 (609h44m31.763970014s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:13:32.136582035 (609h44m32.00766508s) [seg_pos={0 0}, upload=534365]: Begin upload segment... Dec 31 01:13:32.136741648 (609h44m32.007824693s) [seg_pos={0 0}, upload=534365]: Done begin upload segment. Dec 31 01:13:32.136787886 (609h44m32.007870921s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.136952009 (609h44m32.008035054s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137392508 (609h44m32.008475553s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137833196 (609h44m32.008916241s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.137887390 (609h44m32.008970435s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138202520 (609h44m32.009285555s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138443369 (609h44m32.009526414s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138677505 (609h44m32.00976054s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.138997634 (609h44m32.010080679s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139179520 (609h44m32.010262565s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139293667 (609h44m32.010376712s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139647271 (609h44m32.010730316s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.139782598 (609h44m32.010865643s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.140427918 (609h44m32.011510953s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141112661 (609h44m32.012195706s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:32.141783057 (609h44m32.012866102s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Uploading piece... Dec 31 01:13:33.837053250 (609h44m33.708136285s) [upload=534365]: Next returned nil segment Dec 31 01:13:33.837088717 (609h44m33.708171752s) [upload=534365]: Waiting for error group managing segments... Dec 31 01:13:34.123562999 (609h44m33.994646034s) [noise=true , address=145.79.227.7:4293 , piece_id=PAWHOP4S5N2SWCW2NVTGSGGGZDYKVZ52HE5LUXWBC45WTCZ47LXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.124260658 (609h44m33.995343693s) [noise=true , address=145.79.227.20:4438 , piece_id=HHBPYPKJUJOGJA76IYPO3QBCURQBXSZAMDRI6UGEV4RAGKSAZ3FQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.125012440 (609h44m33.996095485s) [noise=true , address=145.79.227.22:4511 , piece_id=UEOO6Y5AFCQ5RPEFKQHNXIM6DQ6CNVGJ46SU7MLECWVP6NEGV4WQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.132053355 (609h44m34.003136411s) [noise=true , address=145.79.227.39:4014 , piece_id=ZB3AA4NG3QF6PLHH6GTGD7PF7FAXUBOEPWVOIOJ5JRQLAZOVBGQA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.135253870 (609h44m34.006336905s) [noise=true , address=145.79.227.3:4150 , piece_id=LRCVVPU4EXBCU6T4FF4PJUUSJYM4RCR5WPVGAYCAWBFIOE77QQRQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.138185082 (609h44m34.009268117s) [noise=true , address=145.79.227.5:4222 , piece_id=AU6TDEXDLGFCXPSFIKYQ3AM4UXBXVMXZ2RVIILMBX6I6UZWU7CJQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.145248080 (609h44m34.016331116s) [noise=true , address=145.79.227.37:4565 , piece_id=YYTOH6DR725IZRPAGNOOTZCSQEI55FEYKDL4PNF3SYONLMOHQ25A, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.146704714 (609h44m34.017787779s) [noise=true , address=145.79.227.2:4113 , piece_id=A2U3D2C7LEPHVPU3IUKBNNM6BFHBW2AOMRRIZ6XPB2MEIANKPRXQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.312085941 (609h44m34.183168976s) [noise=true , address=77.90.31.16:4735 , piece_id=DDIONJAYZBASQJDP3QPISHXACAFZGOW3EBSMIXNNT2LY4DLYXADQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.436213054 (609h44m34.307296109s) [noise=true , address=45.91.116.14:4189 , piece_id=FAPD7HNXB3TV33NFDF4UFJGHY7AWSPTL3KZHMQ4QQIYZ7ZEMSJGA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:34.766896134 (609h44m34.637979179s) [noise=true , address=45.91.116.7:4011 , piece_id=4SKDVZH2AHRVLV5362AOOCGLOBIPEK7WGDBMWLGWZB3Q6NNLQPSA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.034120011 (609h44m34.905203046s) [noise=true , address=77.90.31.21:4877 , piece_id=BBOICMOZ3RSNP3I5XX2AQW3B6SURUFUDCTZG25KXP62FSNOSRQUQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317152980 (609h44m35.188236025s) [noise=true , address=45.91.116.6:4973 , piece_id=IUHID3DDNNPTIAQXX4UCETB5LFFJBYHM7POMA7ALIZYZALICQ73Q, seg_pos={0 0}, upload=534365]: Done uploading piece. err: <nil> Dec 31 01:13:35.317171395 (609h44m35.18825443s) [seg_pos={0 0}, upload=534365]: Segment reached optimal threshold of 13 pieces. Dec 31 01:13:35.317363992 (609h44m35.188447027s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317369673 (609h44m35.188452708s) [noise=true , address=77.90.31.13:4626 , piece_id=5ZZDAE432DNJU4JXKY2THGYXQII4GWWJIRJ65BYF4X2OIHD6IJBQ, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317497957 (609h44m35.188580992s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317503487 (609h44m35.188586522s) [noise=true , address=145.79.227.43:4157 , piece_id=BHS2YBBRH6MD345YZVXB3IMDKAGEWV2NXBKEKVQMYJINLXDEWUNA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317612284 (609h44m35.188695339s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 01:13:35.317643834 (609h44m35.188726869s) [noise=true , address=145.79.227.30:4762 , piece_id=G3O5YLE6DAGDCFNYSVLQ2464XIJRWU2G2MJVTDEBGIWAVY2MRORA, seg_pos={0 0}, upload=534365]: Long tail cancellation due to optimal threshold Dec 31 01:13:35.317682628 (609h44m35.188765663s) [seg_pos={0 0}, upload=534365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:13:35.317778912 (609h44m35.188861947s) [upload=534365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:13:35.317784572 (609h44m35.188867607s) [upload=534365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:13:35.615134704 (609h44m35.486217749s) [upload=534365]: Done uploading segments... Dec 31 01:13:35.615159452 (609h44m35.486242497s) [upload=534365]: Upload finished. err: <nil> Duration: 30.002822792s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.261543666s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:13:42 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.261575477s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.........................................X..........] duration: 28.177976072s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:13:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 28.178002572s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.403227113s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:13:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.403260656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.....] duration: 6.10499026s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:13:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 6.1050164s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.002781883s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:13:03 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.002817261s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.106871469s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:12:41 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.106892981s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.........................................X.........] duration: 34.966114708s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:12:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 34.966160525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.613274869s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:12:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.613302201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX....] duration: 6.113872802s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:12:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 6.113898622s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.455957989s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:12:05 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.455984287s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.627391583s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:11:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.627418112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.........................................X........] duration: 15.507208865s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:11:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 15.507233792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX...] duration: 5.91640484s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:11:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 5.916434547s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.493302312s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:11:30 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.493329182s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.581330897s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:11:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.581356716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.186992327s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:11:02 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.187016692s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.561963271s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:10:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.562027494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........................................X.......] duration: 15.342234982s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:10:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 15.342258707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.147659494s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:10:31 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.147687146s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX..] duration: 6.44768836s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:10:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 6.447715542s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.459328572s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:10:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.459355945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........................................X......] duration: 32.430480581s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:10:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 32.43050698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX.] duration: 6.115982137s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:10:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 6.115997127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..XX] duration: 30.003219478s trace: https://jaeger.dev.storj.io/trace/302875e9b5456820 FAIL (2025-12-31 01:10:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:30.952291082 (609h40m30.823374117s) [upload=534310]: Starting upload Dec 31 01:09:30.952342189 (609h40m30.823425234s) [upload=534310]: Done starting upload Dec 31 01:09:30.952511982 (609h40m30.823595017s) [upload=534310]: Uploading segments... Dec 31 01:09:30.952552770 (609h40m30.823635805s) [upload=534310]: Got next segment. Inline: false Dec 31 01:09:30.952579671 (609h40m30.823662706s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:30.952584220 (609h40m30.823667255s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:31.200060645 (609h40m31.07114369s) [seg_pos={0 0}, upload=534310]: Begin upload segment... Dec 31 01:09:31.200115720 (609h40m31.071198756s) [seg_pos={0 0}, upload=534310]: Done begin upload segment. Dec 31 01:09:31.200160485 (609h40m31.07124353s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.200183600 (609h40m31.071266635s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201328900 (609h40m31.072411945s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.201655632 (609h40m31.072738678s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202001441 (609h40m31.073084476s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202298125 (609h40m31.07338116s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.202753022 (609h40m31.073836067s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203062341 (609h40m31.074145386s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203432997 (609h40m31.074516032s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.203781821 (609h40m31.074864866s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204750115 (609h40m31.07583315s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.204981074 (609h40m31.076064119s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206060129 (609h40m31.077143164s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.206134350 (609h40m31.077217385s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.207278809 (609h40m31.078361845s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:31.208923160 (609h40m31.080006205s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Uploading piece... Dec 31 01:09:32.904529566 (609h40m32.775612611s) [upload=534310]: Next returned nil segment Dec 31 01:09:32.904573228 (609h40m32.775656273s) [upload=534310]: Waiting for error group managing segments... Dec 31 01:09:33.177751323 (609h40m33.048834358s) [noise=true , address=145.79.227.52:4445 , piece_id=O6ZSBU2MB4JQ6AUWMHQVW74F5E7J47QW5WO4T646Q4VLHP2YXYMQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.188115618 (609h40m33.059198663s) [noise=true , address=145.79.227.37:4566 , piece_id=KT7ZA4I33GTHZSICLPHHGJME5QSVDQFI3H6IUZW22RDQSCQDZIGA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.193468411 (609h40m33.064551456s) [noise=true , address=145.79.227.32:4835 , piece_id=ZLUYUXUQCVON5IJCKEQXVAZ5BZMMZSPNN2HNPTYO7P2AZ7M3A6MQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.199886171 (609h40m33.070969206s) [noise=true , address=145.79.227.23:4547 , piece_id=XXKOCFGSLV4MASNGIBJSVKZ5PFRDHWRA55EWMO7BZGCR5K6IRDWA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.203595614 (609h40m33.074678649s) [noise=true , address=145.79.227.8:4329 , piece_id=NH7AM47TX2APQJZNC2NGICACW5QNXPMMWQVCPWXFJXMC65HAAHGQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.205080291 (609h40m33.076163326s) [noise=true , address=145.79.227.15:4874 , piece_id=7ULFRXU43NBIZZDP2UIZEBRHHCYRWBM52VIMKEOK24Q2XVS2KMYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.209178975 (609h40m33.08026201s) [noise=true , address=145.79.227.3:4150 , piece_id=YVAEMQ2OWPAEI45VDFVXZ3P6ECSDGI7UL32THF6MCWMVCEVC4PVQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.217865655 (609h40m33.0889487s) [noise=true , address=145.79.227.38:4978 , piece_id=KPV45YLYDH3RHWEK66VQMTZG57IHBVR62U7Y5RRSNFYKIZENPOJA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.219876184 (609h40m33.090959219s) [noise=true , address=145.79.227.41:4086 , piece_id=HKLPYFM2WBJ5LG2J6TKJIOMQMQYMLQ6R6TER4MLBH3UNYKRK7HYA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.358860787 (609h40m33.229943832s) [noise=true , address=77.90.31.19:4841 , piece_id=WOZHWYYRQGWBKWTZLRA4RM65SB6GP7VHHTZBCAUPMTRLABHC4ZMA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.383597680 (609h40m33.254680715s) [noise=true , address=77.90.31.13:4626 , piece_id=X3DMSHWQ7GNXTLO7CDJL7X4LZF7W2P4SYSX7T4L7UJR6C2W3UAAQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.463911717 (609h40m33.334994762s) [noise=true , address=77.90.31.14:4664 , piece_id=AQFOOWCGCOYVSVEUKPQ7GF2P5UE6XDVSZKLBN4CDUIJIJYLKCTBQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535677719 (609h40m33.406760754s) [noise=true , address=45.91.116.6:4975 , piece_id=DR6AMGJQN7GJXDGCPBJOPID3G2IVM5VFESM4G3RQ7HJ34SULP5AQ, seg_pos={0 0}, upload=534310]: Done uploading piece. err: <nil> Dec 31 01:09:33.535698058 (609h40m33.406781103s) [seg_pos={0 0}, upload=534310]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:33.535862290 (609h40m33.406945335s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.535869223 (609h40m33.406952258s) [noise=true , address=45.91.116.8:4048 , piece_id=QFT3XUVCDP65BZZWJRLJ4MMZANWIROLBJHCXPIMNAAVQSK5X7J5A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.535958804 (609h40m33.407041849s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42100->77.90.31.17:4770: use of closed network connection Dec 31 01:09:33.535974513 (609h40m33.407057548s) [noise=true , address=77.90.31.17:4770 , piece_id=JIO2NPGHUOIVZGDKZK2HQCYUDKM46J7KNECNGRRBWWSNIOWWI5BA, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536069374 (609h40m33.407152459s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:33.536085155 (609h40m33.40716819s) [noise=true , address=45.91.116.14:4191 , piece_id=QRSPTCDJNPLP2JV3YO4NSZXN6OFKJCQ2TAWKXVE6APHOXAQJON7A, seg_pos={0 0}, upload=534310]: Long tail cancellation due to optimal threshold Dec 31 01:09:33.536110973 (609h40m33.407194018s) [seg_pos={0 0}, upload=534310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:33.536258765 (609h40m33.407341811s) [upload=534310]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:33.536265258 (609h40m33.407348303s) [upload=534310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:33.838979262 (609h40m33.710062308s) [upload=534310]: Done uploading segments... Dec 31 01:09:33.839001435 (609h40m33.71008448s) [upload=534310]: Upload finished. err: <nil> Duration: 30.003229977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X....XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..X] duration: 30.003324163s trace: https://jaeger.dev.storj.io/trace/1dba1e705da16f9c FAIL (2025-12-31 01:09:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:09:00.701775153 (609h40m0.572858188s) [upload=534304]: Starting upload Dec 31 01:09:00.701899269 (609h40m0.572982314s) [upload=534304]: Done starting upload Dec 31 01:09:00.702186516 (609h40m0.573269561s) [upload=534304]: Uploading segments... Dec 31 01:09:00.702353704 (609h40m0.573436779s) [upload=534304]: Got next segment. Inline: false Dec 31 01:09:00.702412346 (609h40m0.573495391s) [upload=534304]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:09:00.702419129 (609h40m0.573502174s) [upload=534304]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:09:00.947352402 (609h40m0.818435457s) [seg_pos={0 0}, upload=534304]: Begin upload segment... Dec 31 01:09:00.947418848 (609h40m0.818501893s) [seg_pos={0 0}, upload=534304]: Done begin upload segment. Dec 31 01:09:00.947479895 (609h40m0.81856294s) [noise=true , address=145.79.227.48:4302 , piece_id=ZYLDEARMSOT5PMK5MJUSK24U7VCLNMCSU7I335TI44JNO5GJ7FIA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.947561891 (609h40m0.818644946s) [noise=true , address=77.90.31.19:4844 , piece_id=UJGR7CWYSZ6HJQ7VS5CENXRI5TBYDLNDXBZAT2DE4HM55L77HFCQ, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.948468197 (609h40m0.819551242s) [noise=true , address=145.79.227.31:4797 , piece_id=KGYBJKJT56RUZ7MZLPRRKPPMNIYQTNA2YO55NOFDLKTZ2NFI4DCA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.949348273 (609h40m0.820431308s) [noise=true , address=77.90.31.12:4589 , piece_id=YKFPWHTDJ3O3PAN7XK34VTJOWD5TCNHPQ5643PNGDD7XZH365ETA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.950004232 (609h40m0.821087277s) [noise=true , address=77.90.31.14:4664 , piece_id=J5HDVR7CSBJ65Q454MJXEX4UPOONL2BZZHHA3SBU3PHZCAQGNU5A, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.950456954 (609h40m0.821539999s) [noise=true , address=145.79.227.24:4582 , piece_id=LWGFFVCFW7TPOICNXNCRMQIPH2FQA3NGHQSHZ7IKQTFTTTRDS54A, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.950673286 (609h40m0.821756331s) [noise=true , address=45.91.116.13:4155 , piece_id=UN3M4M3FLQCGHN3D2CZT7FCM3TIGB5L5DIF5RF3CNWULSRXGIP2A, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.951114737 (609h40m0.822197782s) [noise=true , address=45.91.116.12:4228 , piece_id=KCEFSSOX2S3A6NJHLGQKUCH4UBY5ZZYA6PHKD35NCXG2BRJTRISA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.951377396 (609h40m0.822460441s) [noise=true , address=145.79.227.50:4374 , piece_id=OIINHZPTEUD2ZJCZYJLVLQAWVFL3IBECGSDFUILCGZ5BMURY7ZGQ, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.951559633 (609h40m0.822642678s) [noise=true , address=77.90.31.17:4773 , piece_id=WTWJWN7OOHIM3DQSPYDWDBQL3E6PV6HSR6Y33V77JP3CPF3S24CQ, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.951866758 (609h40m0.822949793s) [noise=true , address=145.79.227.26:4655 , piece_id=S7RPRBQO2567YXI25C6VY33CBI7FV2J4ZSL7WLVKKGU35JBTYUMQ, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.958098214 (609h40m0.829181249s) [noise=true , address=45.91.116.6:4976 , piece_id=LCKEICKRRY2NTH2W62SS343SM6WSIEEXMENKDJ5RFCSF5NFOKWWA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.960822531 (609h40m0.831905576s) [noise=true , address=45.91.116.9:4262 , piece_id=IKUOOQVZHMDUVQX246PM35QHUXTFYO2C762D2MK5XCC3O4CRTFBA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.962623200 (609h40m0.833706245s) [noise=true , address=145.79.227.47:4267 , piece_id=MRMAD622MJREQJEGJGSY3ESHFWBE5EHT72NVI445UCIB75UODCNQ, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.964541893 (609h40m0.835624938s) [noise=true , address=145.79.227.45:4231 , piece_id=VMFZDV6OMG2S6DYFEZR2UDDONQ6HW6FKVNP3XPWSZ7PFW3KGBIRA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:00.966244204 (609h40m0.837327249s) [noise=true , address=145.79.227.11:4730 , piece_id=4DBOUI5NKMXEQLIW6LVJKVZNLCYBKWTJMZHPMGIGI5BWLD44P2EA, seg_pos={0 0}, upload=534304]: Uploading piece... Dec 31 01:09:02.664885536 (609h40m2.535968581s) [upload=534304]: Next returned nil segment Dec 31 01:09:02.664905624 (609h40m2.535988669s) [upload=534304]: Waiting for error group managing segments... Dec 31 01:09:03.015637030 (609h40m2.886720095s) [noise=true , address=145.79.227.50:4374 , piece_id=OIINHZPTEUD2ZJCZYJLVLQAWVFL3IBECGSDFUILCGZ5BMURY7ZGQ, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.016352122 (609h40m2.887435167s) [noise=true , address=145.79.227.48:4302 , piece_id=ZYLDEARMSOT5PMK5MJUSK24U7VCLNMCSU7I335TI44JNO5GJ7FIA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.016523729 (609h40m2.887606804s) [noise=true , address=145.79.227.24:4582 , piece_id=LWGFFVCFW7TPOICNXNCRMQIPH2FQA3NGHQSHZ7IKQTFTTTRDS54A, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.050886209 (609h40m2.921969254s) [noise=true , address=145.79.227.45:4231 , piece_id=VMFZDV6OMG2S6DYFEZR2UDDONQ6HW6FKVNP3XPWSZ7PFW3KGBIRA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.054784321 (609h40m2.925867366s) [noise=true , address=145.79.227.31:4797 , piece_id=KGYBJKJT56RUZ7MZLPRRKPPMNIYQTNA2YO55NOFDLKTZ2NFI4DCA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.097354450 (609h40m2.968437515s) [noise=true , address=145.79.227.11:4730 , piece_id=4DBOUI5NKMXEQLIW6LVJKVZNLCYBKWTJMZHPMGIGI5BWLD44P2EA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.115899222 (609h40m2.986982277s) [noise=true , address=145.79.227.26:4655 , piece_id=S7RPRBQO2567YXI25C6VY33CBI7FV2J4ZSL7WLVKKGU35JBTYUMQ, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.119796834 (609h40m2.990879889s) [noise=true , address=145.79.227.47:4267 , piece_id=MRMAD622MJREQJEGJGSY3ESHFWBE5EHT72NVI445UCIB75UODCNQ, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.156087084 (609h40m3.027170119s) [noise=true , address=77.90.31.14:4664 , piece_id=J5HDVR7CSBJ65Q454MJXEX4UPOONL2BZZHHA3SBU3PHZCAQGNU5A, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.161484001 (609h40m3.032567046s) [noise=true , address=77.90.31.19:4844 , piece_id=UJGR7CWYSZ6HJQ7VS5CENXRI5TBYDLNDXBZAT2DE4HM55L77HFCQ, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.169528880 (609h40m3.040611925s) [noise=true , address=77.90.31.17:4773 , piece_id=WTWJWN7OOHIM3DQSPYDWDBQL3E6PV6HSR6Y33V77JP3CPF3S24CQ, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.353124388 (609h40m3.224207433s) [noise=true , address=45.91.116.9:4262 , piece_id=IKUOOQVZHMDUVQX246PM35QHUXTFYO2C762D2MK5XCC3O4CRTFBA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.408731902 (609h40m3.279814947s) [noise=true , address=45.91.116.12:4228 , piece_id=KCEFSSOX2S3A6NJHLGQKUCH4UBY5ZZYA6PHKD35NCXG2BRJTRISA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: <nil> Dec 31 01:09:03.408759785 (609h40m3.27984283s) [seg_pos={0 0}, upload=534304]: Segment reached optimal threshold of 13 pieces. Dec 31 01:09:03.409102829 (609h40m3.280185884s) [noise=true , address=45.91.116.13:4155 , piece_id=UN3M4M3FLQCGHN3D2CZT7FCM3TIGB5L5DIF5RF3CNWULSRXGIP2A, seg_pos={0 0}, upload=534304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:03.409113158 (609h40m3.280196253s) [noise=true , address=45.91.116.13:4155 , piece_id=UN3M4M3FLQCGHN3D2CZT7FCM3TIGB5L5DIF5RF3CNWULSRXGIP2A, seg_pos={0 0}, upload=534304]: Long tail cancellation due to optimal threshold Dec 31 01:09:03.409669688 (609h40m3.280752733s) [noise=true , address=77.90.31.12:4589 , piece_id=YKFPWHTDJ3O3PAN7XK34VTJOWD5TCNHPQ5643PNGDD7XZH365ETA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:56712->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:56712->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56712->77.90.31.12:4589: use of closed network connection Dec 31 01:09:03.409685067 (609h40m3.280768112s) [noise=true , address=77.90.31.12:4589 , piece_id=YKFPWHTDJ3O3PAN7XK34VTJOWD5TCNHPQ5643PNGDD7XZH365ETA, seg_pos={0 0}, upload=534304]: Long tail cancellation due to optimal threshold Dec 31 01:09:03.409964418 (609h40m3.281047484s) [noise=true , address=45.91.116.6:4976 , piece_id=LCKEICKRRY2NTH2W62SS343SM6WSIEEXMENKDJ5RFCSF5NFOKWWA, seg_pos={0 0}, upload=534304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 31 01:09:03.409975440 (609h40m3.281058495s) [noise=true , address=45.91.116.6:4976 , piece_id=LCKEICKRRY2NTH2W62SS343SM6WSIEEXMENKDJ5RFCSF5NFOKWWA, seg_pos={0 0}, upload=534304]: Long tail cancellation due to optimal threshold Dec 31 01:09:03.409998093 (609h40m3.281081138s) [seg_pos={0 0}, upload=534304]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:09:03.410976656 (609h40m3.282059711s) [upload=534304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:09:03.410987187 (609h40m3.282070232s) [upload=534304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:09:03.708184178 (609h40m3.579267904s) [upload=534304]: Done uploading segments... Dec 31 01:09:03.708235224 (609h40m3.579318259s) [upload=534304]: Upload finished. err: <nil> Duration: 30.00335387s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.470140396s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 01:09:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470163461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 4.849477033s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 01:09:17 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 4.849502662s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.........................................X.....] duration: 17.855158393s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:09:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 17.855205562s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........................................X....] duration: 15.469917069s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:08:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 15.469945363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X....XX.XXX....X.......XX.X.X..XXX....X.XX..X.....X...X.X..] duration: 5.956645382s trace: https://jaeger.dev.storj.io/trace/145ba20f8912fb1d FAIL (2025-12-31 01:08:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 01:07:14.778356370 (609h38m14.649439405s) [upload=534277]: Starting upload Dec 31 01:07:14.778408058 (609h38m14.649491093s) [upload=534277]: Done starting upload Dec 31 01:07:14.778560168 (609h38m14.649643203s) [upload=534277]: Uploading segments... Dec 31 01:07:14.778735582 (609h38m14.649818617s) [upload=534277]: Got next segment. Inline: false Dec 31 01:07:14.778779476 (609h38m14.649862511s) [upload=534277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 01:07:14.778783794 (609h38m14.649866829s) [upload=534277]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 01:07:15.030227975 (609h38m14.90131101s) [seg_pos={0 0}, upload=534277]: Begin upload segment... Dec 31 01:07:15.030291315 (609h38m14.901374351s) [seg_pos={0 0}, upload=534277]: Done begin upload segment. Dec 31 01:07:15.030324619 (609h38m14.901407664s) [noise=true , address=45.91.116.8:4045 , piece_id=EUD5AVANNSRFJD5QIMU25FFUI6QFSNKB2CQZ2N5G3SOHVMRFPZCQ, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.030395023 (609h38m14.901478068s) [noise=true , address=45.91.116.12:4226 , piece_id=H7J4A5DMZFBUG27YBZK4K2J77XIWFMW4PEU7YWLE26JITTXCJW6A, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.030877702 (609h38m14.901960737s) [noise=true , address=145.79.227.4:4187 , piece_id=ETK6NLBJSRK6TCIYHXX4MIWM2FVZCOYYEC3C5BOBH3DQBF723DIQ, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.031685961 (609h38m14.902769006s) [noise=true , address=77.90.31.14:4664 , piece_id=4B6OQKRN7SW3XGBXBJM4DVABRH2NWW2YESH4URT6IFYZ3VA7GZSQ, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.032103817 (609h38m14.903186862s) [noise=true , address=145.79.227.45:4229 , piece_id=L77GZBIP5D5CU2GCW2KUNWXRG4NTGJ76LQSZZ6VHPKNTCBGDGVNQ, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.032537753 (609h38m14.903620798s) [noise=true , address=145.79.227.19:4078 , piece_id=ATSMSG4CNHQ537ZKGQWQGJYP7A4T3H5D7IKP747ECIEAF34UVORQ, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.032963723 (609h38m14.904046759s) [noise=true , address=145.79.227.16:4911 , piece_id=Q6MG56SUKZ4RLLDTGGBQPG7CW52XMNYSCE6FDSWK52MAUGKAIKGA, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.034057346 (609h38m14.905140382s) [noise=true , address=145.79.227.42:4121 , piece_id=S2NJ3Z42V5SUE2QQTK6YK2CI6VB2F4CMTH2Y7MK2VOIHPDI5JVLQ, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.034428734 (609h38m14.905511779s) [noise=true , address=145.79.227.52:4446 , piece_id=7OKBQXTRA7S3OP456YB4TXRX4JGY5FU2BGG3VIXVBDRBSQOFAKVQ, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.035184542 (609h38m14.906267588s) [noise=true , address=145.79.227.2:4114 , piece_id=SJFDTIDU3JJRFILRCMBE5W5QFERCMLFKB3XG4UDHDTJ7DSQJDZVQ, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.035956081 (609h38m14.907039127s) [noise=true , address=45.91.116.9:4264 , piece_id=ZG6E4KU6G36BKPUILDBYFBPJJZ7YTWB6O6O5VQ6G6FQKJDYQWN7A, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.036206910 (609h38m14.907289955s) [noise=true , address=77.90.31.10:4518 , piece_id=JIYDN2KWH467K4QVDYCRFRJXMD4LMC2I473ZW4MMXGHNVUNTBW3A, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.037408658 (609h38m14.908491703s) [noise=true , address=145.79.227.33:4871 , piece_id=INEMG4ZPMX2VQBXBLI5S66HJVXTL6UBGTSZYO6RSIVXCG63TZCXQ, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.040499754 (609h38m14.911582799s) [noise=true , address=145.79.227.46:4603 , piece_id=KQXV6CXRAG2MGCBR5W6UPV6U46IEZRNFQYAUA2QJ24N6WQGCL54Q, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.048349100 (609h38m14.919432135s) [noise=true , address=45.91.116.7:4010 , piece_id=Q6N3HJUMTFVYVEPET6GOEYUQPHYNMELXDFVAZULMHS2UTJWZUCVA, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:15.051897335 (609h38m14.92298037s) [noise=true , address=145.79.227.37:4566 , piece_id=LBQIOLWL3UJANZG5VTUFBK3RWS7TNKEVM7TS5KVJM6UJFL6RMUZA, seg_pos={0 0}, upload=534277]: Uploading piece... Dec 31 01:07:16.749311010 (609h38m16.620394055s) [upload=534277]: Next returned nil segment Dec 31 01:07:16.749328584 (609h38m16.620411619s) [upload=534277]: Waiting for error group managing segments... Dec 31 01:07:17.033453302 (609h38m16.904536357s) [noise=true , address=145.79.227.4:4187 , piece_id=ETK6NLBJSRK6TCIYHXX4MIWM2FVZCOYYEC3C5BOBH3DQBF723DIQ, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.036103358 (609h38m16.907186393s) [noise=true , address=145.79.227.2:4114 , piece_id=SJFDTIDU3JJRFILRCMBE5W5QFERCMLFKB3XG4UDHDTJ7DSQJDZVQ, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.036227865 (609h38m16.9073109s) [noise=true , address=145.79.227.52:4446 , piece_id=7OKBQXTRA7S3OP456YB4TXRX4JGY5FU2BGG3VIXVBDRBSQOFAKVQ, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.053394793 (609h38m16.924477838s) [noise=true , address=145.79.227.37:4566 , piece_id=LBQIOLWL3UJANZG5VTUFBK3RWS7TNKEVM7TS5KVJM6UJFL6RMUZA, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.055795905 (609h38m16.92687895s) [noise=true , address=145.79.227.16:4911 , piece_id=Q6MG56SUKZ4RLLDTGGBQPG7CW52XMNYSCE6FDSWK52MAUGKAIKGA, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.062107232 (609h38m16.933190268s) [noise=true , address=145.79.227.46:4603 , piece_id=KQXV6CXRAG2MGCBR5W6UPV6U46IEZRNFQYAUA2QJ24N6WQGCL54Q, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.117947530 (609h38m16.989030575s) [noise=true , address=145.79.227.19:4078 , piece_id=ATSMSG4CNHQ537ZKGQWQGJYP7A4T3H5D7IKP747ECIEAF34UVORQ, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.119479456 (609h38m16.990562521s) [noise=true , address=145.79.227.45:4229 , piece_id=L77GZBIP5D5CU2GCW2KUNWXRG4NTGJ76LQSZZ6VHPKNTCBGDGVNQ, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.214368105 (609h38m17.085451151s) [noise=true , address=145.79.227.42:4121 , piece_id=S2NJ3Z42V5SUE2QQTK6YK2CI6VB2F4CMTH2Y7MK2VOIHPDI5JVLQ, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.223638918 (609h38m17.094721953s) [noise=true , address=77.90.31.14:4664 , piece_id=4B6OQKRN7SW3XGBXBJM4DVABRH2NWW2YESH4URT6IFYZ3VA7GZSQ, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.335261421 (609h38m17.206344456s) [noise=true , address=77.90.31.10:4518 , piece_id=JIYDN2KWH467K4QVDYCRFRJXMD4LMC2I473ZW4MMXGHNVUNTBW3A, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.371196817 (609h38m17.242279852s) [noise=true , address=45.91.116.9:4264 , piece_id=ZG6E4KU6G36BKPUILDBYFBPJJZ7YTWB6O6O5VQ6G6FQKJDYQWN7A, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.389303364 (609h38m17.260386409s) [noise=true , address=45.91.116.12:4226 , piece_id=H7J4A5DMZFBUG27YBZK4K2J77XIWFMW4PEU7YWLE26JITTXCJW6A, seg_pos={0 0}, upload=534277]: Done uploading piece. err: <nil> Dec 31 01:07:17.389321789 (609h38m17.260404824s) [seg_pos={0 0}, upload=534277]: Segment reached optimal threshold of 13 pieces. Dec 31 01:07:17.389495319 (609h38m17.260578354s) [noise=true , address=45.91.116.7:4010 , piece_id=Q6N3HJUMTFVYVEPET6GOEYUQPHYNMELXDFVAZULMHS2UTJWZUCVA, seg_pos={0 0}, upload=534277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 31 01:07:17.389501111 (609h38m17.260584156s) [noise=true , address=45.91.116.7:4010 , piece_id=Q6N3HJUMTFVYVEPET6GOEYUQPHYNMELXDFVAZULMHS2UTJWZUCVA, seg_pos={0 0}, upload=534277]: Long tail cancellation due to optimal threshold Dec 31 01:07:17.389672526 (609h38m17.260755562s) [noise=true , address=45.91.116.8:4045 , piece_id=EUD5AVANNSRFJD5QIMU25FFUI6QFSNKB2CQZ2N5G3SOHVMRFPZCQ, seg_pos={0 0}, upload=534277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 01:07:17.389687676 (609h38m17.260770711s) [noise=true , address=45.91.116.8:4045 , piece_id=EUD5AVANNSRFJD5QIMU25FFUI6QFSNKB2CQZ2N5G3SOHVMRFPZCQ, seg_pos={0 0}, upload=534277]: Long tail cancellation due to optimal threshold Dec 31 01:07:17.389693326 (609h38m17.260776372s) [noise=true , address=145.79.227.33:4871 , piece_id=INEMG4ZPMX2VQBXBLI5S66HJVXTL6UBGTSZYO6RSIVXCG63TZCXQ, seg_pos={0 0}, upload=534277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 31 01:07:17.389699718 (609h38m17.260782754s) [noise=true , address=145.79.227.33:4871 , piece_id=INEMG4ZPMX2VQBXBLI5S66HJVXTL6UBGTSZYO6RSIVXCG63TZCXQ, seg_pos={0 0}, upload=534277]: Long tail cancellation due to optimal threshold Dec 31 01:07:17.389739183 (609h38m17.260822219s) [seg_pos={0 0}, upload=534277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 01:07:17.389853231 (609h38m17.260936276s) [upload=534277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:07:17.389858571 (609h38m17.260941606s) [upload=534277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:07:17.681049295 (609h38m17.55213237s) [upload=534277]: Done uploading segments... Dec 31 01:07:17.681071127 (609h38m17.552154162s) [upload=534277]: Upload finished. err: <nil> Duration: 5.95667073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.........................................X...] duration: 7.065231728s trace: https://jaeger.dev.storj.io/trace/6692771f1d9bd6b2 FAIL (2025-12-31 01:08:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998406888 (609h37m17.869489933s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.998969990 (609h37m17.870053035s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999423213 (609h37m17.870506258s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:17.999952231 (609h37m17.871035276s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000438797 (609h37m17.871521842s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.000949088 (609h37m17.872032133s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.003033838 (609h37m17.874116893s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.004847171 (609h37m17.875930206s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.006308694 (609h37m17.877391729s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.008663187 (609h37m17.879746233s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.011451617 (609h37m17.882534662s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.012760589 (609h37m17.883843624s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:18.014222312 (609h37m17.885305357s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Uploading piece... Dec 31 01:06:19.443835684 (609h37m19.314918729s) [upload=534265]: Next returned nil segment Dec 31 01:06:19.443849530 (609h37m19.314932575s) [upload=534265]: Waiting for error group managing segments... Dec 31 01:06:19.844855896 (609h37m19.715938931s) [noise=true , address=64.49.13.26:4952 , piece_id=EATDKO3RVDVAWPMCMSEJZG7CK45Z55GJMUO66DACQNUOYUYUWVUQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.850883323 (609h37m19.721966368s) [noise=true , address=145.79.227.24:4603 , piece_id=HQSUHCGZAXFCLJPARI4ES5CT73D3GNLJ2UTF2UJ3RRAKYAWWKK6Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.854227971 (609h37m19.725311017s) [noise=true , address=145.79.227.43:4176 , piece_id=CVC3VIERVP6FDWFUUBQ3C63R5JBAOEV75GJMASRV4L6XYG4PDN2Q, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:19.989572655 (609h37m19.8606557s) [noise=true , address=5.231.73.5:4005 , piece_id=LQROKA6VUPJKLIZE2PYJZMHJF52SVYDQIB7YJ5RBBA2I5XULGCXA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.038160412 (609h37m19.909243457s) [noise=true , address=77.90.31.16:4737 , piece_id=5QISDRHH3CIX3GBOZWFGVI5HCRMR3RMZSCC4KINC44ED2RYP77MA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.045562486 (609h37m19.916645531s) [noise=true , address=5.231.73.2:4022 , piece_id=YFH3RAJ35EYRMALO45AQBHFCJO7KIFEHYE3HIAOUED7NZ5AAVVAA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.110196167 (609h37m19.981279202s) [noise=true , address=77.90.31.17:4804 , piece_id=3TYHHBI5PBRBNJMJYZNIL3MSKX7JFIQUWGPZRXUXCUKOJLK5MZUA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.180265638 (609h37m20.051348694s) [noise=true , address=45.91.116.6:4002 , piece_id=5BENLC3SQHUHMN6EZN47GBMUBMERR74MPQF4YCGSMDGYIKGC7PBA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.193916461 (609h37m20.064999506s) [noise=true , address=45.91.116.8:4077 , piece_id=OAMG33FNNBKQAOXCGORIU7LQGKD2EPH64PA33FCS7JC64KICQREQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.724980742 (609h37m20.596063777s) [noise=true , address=77.90.31.14:4696 , piece_id=AFRK6SBDUSTSWBG22IU7CVTIT774PENFKYOIKTTG42CI5IBQSPLA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.767733068 (609h37m20.638816103s) [noise=true , address=145.79.227.30:4782 , piece_id=R2CNB63KEQFO2FVF2DH3LOINRIWKDHQICDSXJZAUG2X4CF2VXIQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.809884670 (609h37m20.680967715s) [noise=true , address=145.79.227.44:4219 , piece_id=ILJTHSDETVN273ML35ABI5ERFLORQGUK7YVCFCMKUSL6OPBC2PDA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.904592756 (609h37m20.775675791s) [noise=true , address=145.79.227.33:4882 , piece_id=L24XKUJFZYFDUJ4EELNUE4EX2BCRGUKJIY6MIYGEPU7GMCQ2CFIQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977540337 (609h37m20.848623382s) [noise=true , address=145.79.227.49:4352 , piece_id=SV2NVRBAHOSSCIMBXHK4TXZFWP625CLQ3WAJ4F4LJ6BAWQTRWJQQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: <nil> Dec 31 01:06:20.977565255 (609h37m20.84864829s) [seg_pos={0 0}, upload=534265]: Segment reached optimal threshold of 14 pieces. Dec 31 01:06:20.977775716 (609h37m20.848858761s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45066->145.79.227.6:4283: use of closed network connection Dec 31 01:06:20.977784612 (609h37m20.848867647s) [noise=true , address=145.79.227.6:4283 , piece_id=ECTZJSRT5AHVBETWBIUFPTZ64X44LUWGFAJ57JVCJMGDPNGIOKGQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.977846340 (609h37m20.848929385s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n): protocol: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44192->145.79.227.21:4490: use of closed network connection Dec 31 01:06:20.977858483 (609h37m20.848941528s) [noise=true , address=145.79.227.21:4490 , piece_id=FTZML6R5TWPW54AJB6ZHXECXK5POQLZBNQH6KP7RZGOYM5XBIGDQ, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978094993 (609h37m20.849178038s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF): protocol: expected piece hash; context canceled; EOF Dec 31 01:06:20.978105122 (609h37m20.849188167s) [noise=true , address=77.90.31.13:4638 , piece_id=7BNCYQC3NWXO3CRAIRA35ZRFKXTHDRVLAAN5XWO32EHGEGYDCLVA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978136983 (609h37m20.849220018s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48490->64.49.13.169:4708: use of closed network connection Dec 31 01:06:20.978148856 (609h37m20.849231901s) [noise=true , address=64.49.13.169:4708 , piece_id=I3K7RPERZKF7HLCXJVT3QGHQKGPZO62GPTEWEVLUQI3WKXP5PMQA, seg_pos={0 0}, upload=534265]: Long tail cancellation due to optimal threshold Dec 31 01:06:20.978171629 (609h37m20.849254674s) [seg_pos={0 0}, upload=534265]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:06:20.978256441 (609h37m20.849339476s) [upload=534265]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:06:20.978263184 (609h37m20.849346219s) [upload=534265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:06:21.275269351 (609h37m21.146352406s) [upload=534265]: Done uploading segments... Dec 31 01:06:21.275294678 (609h37m21.146377713s) [upload=534265]: Upload finished. err: <nil> Duration: 7.065250275s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)